RECONFIGURABLE INSTRUMENT PANELS 
Cross Reference to Other Applications 

[0001] This application claims benefit of Provisional application U.S. Provisional 
Application No. 60/393,130 filed on July 3, 2002 entitled RECONFIGURABLE TACTILE 
CONTROL DISPLAYS FOR VEHICLE INSTRUMENTATION AND CONTROL and U.S. 
Provisional Application No. 60/458,434 filed on March 31 , 2003 entitled VEHICLE 
INSTRUMENTATION AND CONTROL USING RECONFIGURABLE TACTILE 
CONTROL DISPLAYS. 

[0002] This application is a continuation in part of my co-pending applications No. 
09/435,854 filed on November 8, 1999 "Touch TV and other Man Machine Interfaces" 
and 09/ 789,538 filed on February 22, 2001 . "Programmable Tactile Touch Screen 
Displays and Man-Machine Interfaces for Improved Vehicle Instrumentation and 
Telematics", the disclosures of which are incorporated by reference in their entirety. 

Cross reference to related applications bv the inventor 

[0003] 1 . Programmable Tactile Touch Screen Displays and Man-Machine 
Interfaces for Improved Vehicle Instrumentation and Telematics, SN 09/ 789,538; 
[0004] 2. Touch TV and other Man Machine Interfaces, SN 09/ 435,854 a 
continuation of application Ser. No. 07/946,908, now US Patent 5,982,352, which was 
a continuation-in-part of application SN 08/290,516, filed Aug. 15, 1994, now US Patent 
6,008,000; 

[0005] 3. More Useful Man Machine interfaces and applications SN 
09/433,297; 

[0006] 4. Picture Taking method and apparatus SN 09/568,552; 

[0007] 5. Methods and Apparatus for Man Machine Interfaces and Related 

Activity SN 09/568,554; 

[0008] 6. Useful Man Machine interfaces and applications, SN 09/138,339; 
[0009] 7. Camera Based Man -Machine Interfaces SN 09/612,225; 
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[001 0] The disclosures of the above U.S. patents and co-pending patent 
applications are incorporated herein by reference in their entirety. Also incorporated by 
reference is the disclosure of USP 6,314,631, 6,044,183, 5,510,625 and 4,394,683 by 
the inventor and colleagues . 

[001 1] Federally sponsored R and D statement - not applicable 
[0012] Microfiche Appendix- not applicable 
[001 3] Federally sponsored R and D statement - not applicable 
[0014] Microfiche Appendix- not applicable 

FIELD OF THE INVENTION 

[0015] The invention herein continues from my copending applications and 
patents referenced above, and primarily concerns Vehicle Instrument Panels (also 
called an "IP", and in some cases a dashboard, or dash panel or dash), based on what I 
call a "Reconfigurable Tactile Control Display" (or RTD for short) to provide a wide 
range of information, and the safe input of data to a computer controlling the vehicle 
subsystems. The invention makes use of a unique tactile form of touch screen 
employing physical selection or adjustment control means which can be felt in a tactile 
sense, and typically utilizes electro-optical transduction of knob position, finger location, 
and other variables at multiple points at once. A revolutionary, yet familiar, large 
screen form of instrument panel enabling safer vehicle operation results which is simple 
to operate, stylistically attractive, customizable byjthe user, and programmable in both 
the tactile and visual sense. It is also thought to be lower in cost than most 
conventional instrument panels and may have applications in other fields such as ; 
military vehicles, control systems, home automation systems, CAD or Graphics 
terminals, and the like. 
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BACKGROUND OF THE INVENTION 

[0016] The Automobile Dashboard or Instrument Panel (IP) as we know it today 
has evolved in the last 60 years or so, into a combination of control details such as 
knobs, switches, and sliders for actuation and selection of functions, together with 
analog or digital displays and light based tell-tales for communication of information to 
the driver. The actual provision of such components is governed in the USA by FMVSS 
(Federal Motor Vehicle Safety Standard) 101: Controls and Displays. In general, this 
standard specifies overall ground rules for the more critical functions operated by hand 
by the driver, leaving non-critical functions such as entertainment to the option of the 
manufacturer. Even some non-critical controls however, are specified as to their design 
goals, to avoid driver confusion. A particular stipulation of FMVSS 101 states that the 
Labelling (written or pictograph) for controls must be on or adjacent the control. 
[0017] The Vehicle Instrument Panel is constrained as to available space for the 
instrumentation and control functions. Controls used by the driver cannot be so high as 
to obscure the drivers vision, or too low so as to require too much angular diversion of 
ones line of sight while in motion (the SAE guideline is 30 degrees max from normal line 
of sight out the windshield ). 

[0018] In a right - left, or cross-car context, the hand operated controls cannot be 
significantly behind the steering wheel used in present day vehicles, or too far to reach 
over to the passenger side by even the smallest driver using them. 
[0019] In order to achieve added functionality made possible by modern 
electronics while at the same time reducing the ever increasing clutter of further controls 
required to achieve even present day functions, there is a big incentive to develop 
Instrument Panels which can fit in the available instrument panel space yet be 
reconfigured to serve multiple purposes, always with safe operation paramount. In 
particular, there is a need to provide such IP's which can not only be used on less 
critical control and informational functions, but for primary control functions as well, 
reducing cost and complexity and increasing justification. 

[0020] Examples of prior art directed at some aspects of the problem are: 
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[0021] US Patent 6,246,935 by Buckley , June 12, 2001 Vehicle instrument 
panel computer interface and display; 

[0022] US Patent 6344793 Process for assisting a user of a motor vehicle. 
Geek et al , assigned to Daimlerchrysler; 

[0023] US Patent 5,757,268 Toffolo , et al. May 26, 1 998 Prioritization of vehicle 
display features (on a reconfigurable display), assigned to Lear Corporation; 
[0024] US Patent 6,373,472, Driver Control interface System, Palalau et al, 
assigned to Lear Corporation; 

[0025] US Patent 5,539,429 Yano , etal. July 23, 1996 Touch device panel, 
assigned to Mitsubishi; 

[0026] US Patent 6,067,081 Hahlganss , et al. Method for producing tactile 
markings on an input surface and system for carrying out of the method , assigned to 
VDO; 

[0027] US Patent 5,956,01 6 Kuenzner et al Operating device for Menu controlled 
functions of vehicle, Assigned to BMW; 

[0028] US Patent App 09/963,565 April 4, 2002 by Hirose, et al. Display device 
with screen having curved surface , assigned to Nissan; 

[0029] PCT Patent application PCT /GB99/04006, Touch Sensitive Switch, 
Butler et al, assigned to Ford Motor Co.; 

[0030] US pat app Kind Code 20020002432 Bockmann et al. filed January 3, 
2002 assigned to Volkswagen. Automobile multifunctional display and control device 
method; 

[0031] In terms of the control of the vehicle, there are basically three major 
components (not including foot based controls). 

[0032] Data Displays by instruments either analog or digital, and accessories. 
Vehicle data is generally speed, fuel level , engine rpm, battery charge:, etc. Someof 
these are rapidly varying (e.g. RPM) others slow (eg. fuel). Some are more important 
than others. 

[0033] Hand operated controls, generally today in the form of knobs , switches, 
sliders, levers and dials. Switches and knobs are most common today it seems. Most 
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are located on the instrument panel, but some are on stalks protruding from the steering 
column, and window and seat controls and outside mirror controls can most often be 
found on the doors, or seats. Some controls (such as cruise control, auxiliary radio 
controls etc) can be found on the steering wheel in many vehicles. 
[0034] Control labeling , today nearly universally achieved with printed data 
(language or pictograph) on or adjacent the control, to comply with FMVSS 101 . In 
several cases, the data cannot rotate with the knob. 

[0035] Modern technology has the ability to greatly expand the amount of data 
which can be provided, as long as the means exists to display it safely (legibility, 
position, etc). However, the means to interact with the data, if required or desirable has 
to be present in the controls, which at the same time need to comply with regulations. 
The steering wheel, which is a logical place for more control functions, is pretty much 
constrained today in this regard by the presence of the airbag, and the requirement for 
slip rings or other complex wiring 

[0036] Central to the issue, is the question of just how the driver uses the 
controls?. I have observed that a driver of a vehicle today, interacts with the controls of 
the vehicle instrument panel, primarily in three ways: 
[0037] By sight only; 

[0038] By touch /feel only (possibly accompanied by a sound, such as a click- 
usually corresponding to a feeling sensation such as provided by a detent ; 
[0039] By a quick glance, and then touch - possibly followed by another quick 
glance to check a further or final setting of a knob, slider control, etc. 
[0040] Feedback as to correctness of the action taken, if not obvious from sight 
or feel of the controls, is often times derived from the action of the device controlled 
itself, for example seeing and hearing the windshield wipers moving at high speed, 
when they were moving at low speed at a previous control setting. 
[0041] Often while driving, only methods 2 and 3 above can be effected, as the 
control details may be too small to read or operate (radio function buttons are notorious) 
and/or require too much concentration (i.e. too many glances at the controls and their 
lettering, pictographs, and position) while driving to allow one to take ones eyes off the 
road for the time required to use sight only. Accordingly, for many users today, some 
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functions are effectively inoperable while driving, since they cannot be worked by touch 
only due to the crowded nature of their placement, and they are too hard to see in a 
glance- especially for those who cannot easily correct their vision away from the far 
sighted vision needed to drive. 

[0042] It should be noted that certain technical papers indicate that single 
glances of more than 2 seconds duration are considered unsafe, and some traffic safety 
researchers feel the limit to do any control activity should be just a few glances of no 
more than 1 .2 sec. each. These studies indicate in considerable detail that excessive 
glance time can be related to ten's of thousands of deaths per year in the USA alone. 
[0043] The new Auto Industry guide lines for "Glance time" suggest that control of 
the vehicle function should not require one to take ones eyes off the road for than 2 sec 
max, and no more than 10 glances (of 1-2 sec. ea) should be required to complete the 
operation in question, (while seemingly are a large number, some navigation tasks in 
certain vehicles have historically taken considerably more than this). This would seem 
the minimum stipulation one could possibly recommend in light of the suggestions in 
many technical papers. And it suggests that indeed there is considerable room for 
improvement to not only the situation today, but even what is being contemplated. 
[0044] Not only are there problems seeing or touching small complex controls , 
some controls are not intuitive, and hard for the general public to understand in the 
manual, even if one has time to read and comprehend it. This manual comprehension 
problem is increased considerably, for those buying more expensive vehicles on short 
term leases. These vehicles are in addition, usually the ones with the most features, 
(and generally therefore the smallest buttons, due to instrument panel congestion), 
and many persons seldom take the time to fully digest the manual. . (Some manuals 
today for feature oriented luxury cars are several hundred pages long !) 
[0045] Generally a few functions, such as tu/n signal stalks, transmission levers, 
and some knobs are big enough and/or simple enough such that only touch is needed, 
combined with feedback from sensing the physical action enabled (e.g. wipers switched 
to high speed). Thus after some acquaintance period, most vehicles enable the driver 
to perform the basic functions by touch alone, or just a reasonably quick glance and 
then touch. However, this often applies only to basic operations, given the learning 
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curve involved and the difficulty in operation of added features, such as complex HVAC 
(heating ventilation and air conditioning) systems, navigation systems, and 
Infotainment" devices. 

[0046] The present situation thus greatly constrains the addition of still further 
vehicle related functions, made possible by computing, electronics and communication 
technology. Heretofore, the addition of such functions in some cases has been 
distracting to the driver and thus dangerous to the public. With fixed instrument panel 
space, much of which is taken up by airbags, glove compartments and the like, 
manufacturers have resorted to the above mentioned expedient of ever smaller and 
more complex controls in the remaining space, in order to gain functionality. Many are 
in addition located out of the drivers normal line of sight, and some even require one to 
stare at changing numbers in an awkward location. Others require reading very small 
print (often impossible for those requiring reading glasses while driving - a real dilemma 
for many drivers over middle age particularly). 

[0047] Of late, many have shown interest in "Voice Recognition" as an answer to 
such problems, and increasing numbers of vehicles such as the Infiniti Q45 and Jaguar 
"S" model, have such incorporated. Using ones voice (and the listening to computer 
voiced data streams) in theory could reduce the need for glancing at all. 
[0048] However, voice recognition has many problems. For example, to be 
useful, vocal prompts for the driver are often needed, which can be time consuming, 
frustrating, and difficult to hear. Undue concentration on same can also cause unsafe 
driving, and masks the presence of sirens. In addition, voice is both linear, and 
sequential. Only one message/action can be executed at once, and generally needs to 
follow a previous one. And the whole message has to be understood by the driver, or 
data can be lost and the process required to be repeated, which can cause driver 
frustration and endanger safety in extreme circumstances. 

[0049] I have felt for some time that voice is not the answer. A recent study by 
Dr. Ben Shneiderman at the Human Computer Interaction Lab at the University of 
Maryland reported in The Washington Post, Thursday May 9, 2002, noted that voice 
commands also require much more precious brain activity than visual "eye-hand" tasks. 
Thus voice cannot be good for driving safely. The Instant invention however, 
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maximizes the efficiency of such Eye -hand tasks, making them familiar and easy to 
see and act upon. 

[0050] Voice aside, perhaps the only way more data can be presented to, or 
received from, the driver is through the use of specialized displays and entry devices 
which can be reconfigured to suit the need of the user at the instant of use. Such 
displays are commonplace today in the computer world (e.g. "Windows", "Web 
Browsers"), so why not in a car? 

[0051] Some companies have proposed or developed conventional 
reconfigurable computer "windows" type systems using up/down/left/right buttons or 
even joysticks to select the various software presented screens and menus thereon in 
an attempt to solve this problem. Such devices are however, not intuitive and hard to 
use, for at least 95% of the populace I feel. Because of such problems, even where 
implemented, they have been relegated to non-critical functions, such as navigation, 
climate control, cell phone dialling or audio system entertainment. And even here, they 
are intrinsically unsafe in many manifestations. There is little or no tactile feel, and too 
much reliance on consecutive visual recognition of displayed items- which in addition 
are often small in size, as noted above. 

[0052] Problems with computer displays of the multiple windows type used in 

vehicles, are discussed in USP 5,995,104 by Kataoka, aimed at a Navigation system. 

Some opportunities available with computer based data presentation in visual and 

multimedia form are discussed in Obradovich, et al USP 6,009,355. 

[0053] It should also be noted that the lack of natural intuitive tactile feel further 

makes such menu driven "pure computer screen" type systems slower to operate and 

constantly requiring visual concentration. This is an issue when the control system is 

used for any time-critical tasks and has obvious safety connotations. — 

[0054] This is not just true in cars, but in other stressful situations as well, for 

example in military vehicles, construction equipment, and control of home appliances 

by a harried housewife in the kitchen. 

[0055] Another approach to reconfigurability is a touch screen , which is widely 
regarded as one of the most intuitive of computer interfaces. To my knowledge, the 
application of touch screens for computer input in automobiles was first achieved in the 
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1988, in the General Motors Buick Riviera using a relatively small CRT display located 
in the center stack, in a relatively low position. Such a display is described in USP 
4,787,040 by Ames, et al. entitled Display system for Automotive Vehicle issued to IBM 
Corporation November 22, 1988 . 

[0056] However, widespread employment of such touch screens has historically 
has been limited because it cannot be operated by feel, and requires of a driver more 
than a quick glance to operate successfully. This is in part I believe because there is 
nothing on a conventional touch screen to reference ones self to, except the displayed 
data itself - which then has to be continually read. 

[0057] To combat this problem (experienced in the early Buick display and 
others), some vehicles such as the Cadillac CTS have incorporated ordinary non-touch 
screen LCD displays, with buttons and other conventional touch type controls around 
the periphery thereof to effect reconfigurable operation in conjunction with icons 
displayed on the screen nearby (and thus to a degree meet FMVSS 101 ). Taking this 
a step further, the Denso Company has worked with Jaguar to introduce a touch screen 
of the LCD type in the their X model. It has a small 7 inch screen, also surrounded by 
conventional buttons and touch controls, and located in the center stack. But the 
display itself has no reference and no feel, and the surrounding buttons are, as in many 
other examples, small and hard to read. Thus the fact that certain touch icon functions 
on the screen are reconfigurable is only a small improvement on the previous and 
prevailing situation. 

[0058] Another example of a touch type data entry device is that of 6,067,081 by 
Hahiganss, et al., assigned to VDO corporation of Germany, an Automotive parts 
manufacturer. This is a device providing programmable sensations, which can be used 
in conjunction with a display screen, though there is no teaching that the touch is of the 
display screen itself, desirable for intuitive interaction. Thus the Hlahlganss et al 
invention would seem not to comply with FMVSS 101. 

[0059] Palalau et al, USP 6373472 includes switches on>the steering wheel with 
a heads up display on the windshield. This is a novel approach which offers many 
advantages, but requires added cost and complexity, and a departure from the 
customary vehicle operation that drivers are used to. In addition it could be dangerous 
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in some cases as the more fumbling with the wheel that occurs, the more the possibility 
of changing vehicle direction as a result. And too, the wheel, with its airbag and rotary 
motion, severely limits the amount of sophistication one can put in the controls thereon. 
[0060] A recently published PCT application PCT /GB99/04006 by Butler et al of 
Ford motor company illustrates a multifunctional switching device for the operator to 
use whose groove aspect which bears some similarity to certain tactile groove aspects 
of the invention, though it is neither a touch screen nor reconfigurable per se. Like 
some embodiments of the instant invention, it can provide data only when needed by 
the user ("secret until lit" feature), but it cannot be reconfigured as to what this data is. 
[0061] An alternative control device for a car having sensations of feel is a force 
feedback joystick coupled with a conventional (and programmably reconfigurable) 
display, such as an LCD based display, on the instrument panel. One example, called 
"l-Drive", has been introduced by BMW in its 7 series car and appears to be disclosed 
at least in part in 5,956,016 of Kuenzner et al of BMW and in US Patent 6,154,201 
assigned to Immersion Corporation, which describes various force feedback of a 
conventional type to a knob useful for automotive or other use. This has some similarity 
to certain " programmable feel" related aspects of the present invention and its 
predecessors, even though the knob disclosed is not located on the screen, nor inter- 
related with touch characteristics of the screen as is the instant invention. 
[0062] As a general comment, l-Drive does not appear to be intuitive to use, and 
has received considerable criticism in the Automotive press (see for example, Road and 
Track, issue of May 02) The manufacturer must have realized this, as in the 7 series 
manifestation at least, many of the functions are backed up with conventional controls, 
as well, thus adding cost. This also may be because the FMVSS 101 regulations 
require the markings of a control to be on or adjacent the control, and the joystick 
device by itself cannot meet this criteria. 

[0063] If one can provide a reconfigurable instrument panel that can be safely 
used, then US Patent 5,757,268 Toffolo , et al. (referenced above) gives a good 
description of some of the opportunities available in optimizing the presentation of 
displays and the information provided. Toffolo however, is focused on a conventional 
display screen approach and is generally limited relative to the display size, two areas 



10 



over come by the instant invention. And Toffolo does not address several control areas 
of considerable importance , such as designation or confirmation by the driver of data, 
or the overriding of important controls (e.g. speedometer) by visual or other data in the 
case of emergency situations . 

[0064] Finally there are many projects around the globe aimed at sensory 
equipped intelligent vehicles and highway systems. Representative patent applications 
in this area are 09/963490 Road lane marker recognition by Furusho et al, and 09/ 
951499 Lane recognition apparatus for vehicle .Shirato et al, both of which are assigned 
to Nissan, and utilize TV camera and machine vision technology to help guide a car 
such that it stays within lanes. Some of this technology however is very difficult to 
perfect in a full automatic mode. I know of no reference which has addressed the issue 
of manual assist to such systems, provided in a large screen tactile control and display 
device of the type disclosed herein. 

[0065] To conclude, no prior art reconfigurable instrument panels known to me 
provide any feel on a display screen itself, as to where on the screen the users finger is, 
nor do they provide a method for tactilely signalling information back to the user data 
which would make reliance on long and dangerous glances unnecessary 
[0066] In addition, no prior art instrument panel uses common knobs and other 
selection and control details familiar to the driving public of today which can be 
reconfigured either physically or in terms of their displayed function, data or other 
variables in a manner that appears to intrinsically comply with FMVSS 101. 
[0067] Furthermore, no prior art instrument panel I am aware of provides means 
for safely displaying real time or down loaded video data to the driver in a large enough 
to easily to see in a cost effective manner, vital for observation of critical events inside 
and outside the vehicle, as well as to make downloaded information from remote 
sources easy to see at a glance. And in the same vein, no known instrument panel is 
capable of providing large lettering for all necessary controls to aid* the elderly or vision 
impaired driver. 

[0068] And in the same vein, no prior art reconfigurable instrument panel to my 
knowledge, and even many controls of conventional IP's, do not utilize controls which 
can easily be "hit" or otherwise actuated, with just a helping glance or by feel alone. 
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And, no prior art teaches incorporation of both knobs or other control details and tactile 
touch functions on an instrument panel to allow one to optimally execute functions in 
concert. 

[0069] In addition to the prior art referenced above, there are a series of related 
US Patents by Denny Jaeger, some of which are in conjunction with Kenneth Twain (eg 
USP 5572239, 5841428, 6326936) which relate to techniques for implementing control 
devices such as knobs and switches on flat panel displays (electroluminescent or LCD) 
to achieve the benefits of controls function reconfiguration. The approach taken 
however, results in complex (and expensive) systems which, while programmable as 
to the display, do not facilitate interchange of the physical components. In many 
embodiments too, they require significant optical compromises in the display (such as 
blank spots) and the flat panels used require specialized busbars removing these 
displays from the mainstream activity. In some cases, intolerable leakage problems can 
occur as the LCD liquid crystal material can leak around the knob shafts for example. 
[0070] To solve the problems encountered, some embodiments (of the many 
shown) of Jaeger and Jaeger et al., have attempted to use optical techniques to shoot 
through the screen so to speak . In the earliest example, a detector and source for 
each device such as a knob or switch, is located and attached to the other side of the 
display. It is positioned to shoot through either a specialized transparent busbar or a 
bus bar routing hole specially created in the display. This can work presumably, but is 
complex and expensive, and would appear to use up valuable display space, as well as 
requiring a device dedicated and positioned to operate each physical control. In LCDs 
requiring separate specialized backlight sources for their operation, such sensing 
devices would appear impractical, as they would block the radiation from the backlight. 
[0071] Where optical sensing is employed in later patents by Jaeger and Jaeger 
et al, these inventors have apparently focused on having the sensing device (eg a 
phototransistor) on the front of the display, located on a member attached to the screen 
in some way. Either the light source is also on the front, or light from the display is 
programmed to interrogate the position of the sensor on the knob which senses this 
activity. The inventors also have disclosed sensors of this type which are largely 
incremental, rather than absolute, which limits some control regimes and can cause 
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errors. Additionally, Jaeger has disclosed a small LCD device electrically placed on the 
front of a knob or button and controlled to change information thereon, an approach 
which requires specialized wiring and LCDs for every physical control detail used. 
[0072] Finally, Jaegers invention are specific to flat panel displays and cannot 
provide curved or irregular display capability of maximum utility and style for Automobile 
instrument panels, and teach little or no ability to use materials other than glass as the 
display surface, also a requirement in vehicles (for passive safety and other reasons), 
especially as the displays grow in size. 

[0073] The instant invention solves all of these problems, by using an electro- 
optical detection means, typically an inexpensive TV camera, to sense, typically from 
the rear of a rear projection display of nearly any material and shape, more than one 
physical detail and its position, and generally its absolute position as well. And it can 
operate on any desired combination of physical control details (such as knobs sliders 
etc) with only a software change. This then enables the complete interchange of the 
display and control surface, including any control details such as knobs, which are 
simple passive devices with no wires needed. In addition, the invention herein, unlike 
the prior art, allows an integrated touch screen capability to be provided which can be 
completely integrated with the physical control capability, in some cases even at no 
added manufacturing cost. The Instant invention is thus arguably of higher 
performance, simpler to maintain, and lower in cost- especially critical for high volume 
applications such as automobiles. 

Summary of the Invention 

[0074] Since Voice based systems cannot be relied on for command and control 
functions, some combination visual and tactile system is needed, resembling to a 
degree the instrument panel today, which also has the advantage of ease of learning 
and compliance with FMVSS 101 if well designed . But how to make this reconfjgurable 
to allow added functionality and other features? And can it be done in a way that could 
improve the safety of operation of vehicles over conventional practice today? 
[0075] A touch screen approach becomes attractive, if it could be envisioned it as 
both a tactile and a visual replacement for a large measure of the automobile user 
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instrumentation and control functions of the instrument panel of today - a dramatic 
move, filled with other advantages in flexibility, ease of use and user benefits, as well as 
potentially lower cost. If such displays could be physically larger, and operated at least 
in part by feel like a conventional instrument panel, then they would be much more 
acceptable in the car, for use by the driver. And they could thus perform more of the 
required control functions of the vehicle as well as telematics, communication, and 
other activities. And they could potentially comply with FMVSS 101 and other existing 
regulations. 

[0076] However, as described above, no prior art touch screen type device has a 
"feel" or "tactile" quality to enable use by a driver of a vehicle without undue visual 
concentration. There is no non-visual cue, either active or passive, in the prior art 
touch screen display to indicate where to touch, or what has been touched. One has to 
physically study the display to obtain this information, which may be difficult or 
impossible in certain driving conditions. 

[0077] The invention herein answers this need with a new form of "touch screen 
like" control device particularly suited for Automotive instrument panel application. It 
introduces the concept of a "Control Surface" which together with the associated display 
creates what I have called a "Reconfigurable Tactile Display" (or "RTD"). As 
explained further herein, it can be much larger than a conventional instrument panel 
display as it can much more efficiently utilize the available instrument panel "real 
estate". Because of this it is further capable of displaying video, textural, or graphical 
images in a manner which can be acted on easily by the driver of the vehicle. The RTD 
invention not only has a number of quasi-conventional and other tactile characteristics 
that make it ideal for instrument panel usage in vehicles, it also may find application in 
aircraft and certain military control applications, as well as in the home. 
[0078] The disclosed invention, is similar in .certain key aspects to the 
conventional instrument panel of today (for example in its use of quasi conventional 
knobs, sliders and switches), and appears to comply with the US National Highway and 
Traffic Safety Administration Federal Motor Vehicle Safety Standard 101 for Controls 
and Displays (FMVSS 101). It may be one of the few, and perhaps the only, fully 
reconfigurable device capable of doing so, an extremely important feature. And it easily 
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meets and exceeds the recent industry promulgated guidelines in so far as glance 
related Driver Distraction issues are concerned . See for example, Automotive News, 
April 5, 2002, "Automakers set Telematics guidelines - Alliance (of Automobile 
Manufacturers) seeks to limit driver distraction". 

[0079] The disclosed invention, including co-pending applications incorporated by 
reference, contains unique embodiments which allow one to interact, by feel, as well as 
by sight and voice/hearing, with displayed data . It encompasses four main areas, 
generally, but not necessarily, employed in synergistic combination: 
[0080] 1 . A display including several features commonly associated with a 
touch screen, but in a new and greatly expanded form which can be sensed by the 
driver or others in several tactile manners, as well as visually. This allows the display to 
double as a control surface, and in preferred embodiments the tactile sensation 
indicates where to touch, but does not appreciably disrupt video or other data which 
may be displayed in the same area of the screen. One can with this feature, find the 
appropriate place to touch with only a very brief glance at the screen. And in some 
cases by one can determine this by feel alone, using the programmable feedback 
aspects of the invention, or if the user has learned by previous use, the location of 
certain tactile points in relation say to the sides of the screen, or to knobs or other 
features on the screen. 

[0081] 2. A tactile, physical, selection or adjustment means, (also called 
herein a control detail), such as a knob, slider, lever, or switch, programmable in its 
tactile or visual nature, and generally (but not necessarily) incorporated with, and 
generally operated in conjunction with the touch screen just described. In a preferred 
embodiment one or more knobs, sliders or other details are located right on the screen, 
such that the representation of their meaning or magnitude can be reconfigured by 
programming the display and the associated readout of their angular or Cartesian 
position on the screen . 

[0082] 3. An optional programmable touch sensation generator as well as 
means for generating audio feedback as desired, operable with either 1 or 2 above, 
which further assists "eyes on the road" driving while utilizing a programmably variable 
control and display device. This allows the driver to determine the state of controls, 
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and even what to do with the control, with little or no visual data needed in some cases, 
thus greatly enhancing utility of electronic and other features which may be provided to 
the driver using modem camera and computer technology, for example. . In several 
embodiments, a force, vibration, pulsation, or other sensation felt may itself be 
programmably changed, adding to driver understanding, and enhancing safety. 
[0083] 4. In the preferred rear projection embodiment, the above features 
may be cost effectively provided in an aesthetically pleasing large display/control 
surface of maximum size in the available "real estate" on the instrument panel within 
easy vision and control of the driver. This large display/control surface provides the 
ability to display and interact with much more data, in a much more visible and effective 
manner. Besides allowing the driver to optimally control the vehicle functions , video 
and graphic data can also be safely provided, for example, from sources within the car 
(such as children in car seats, or mechanical conditions, such as power or braking 
distribution, engine data etc), on the exterior of the car (such as obstacles, rear view TV 
images or curb locations), or transmitted to the car from remote sources, such as 
"Onstar" or TV cameras located at intersections along the roadway. 
[0084] It should also be noted that the large display afforded by the invention, 
and its associated visibility to the driver (and especially older or vision impaired drivers) 
is further synergistic with some of the safety and ease of use features of the invention. 
The degree of synergy provided is far more than just "bigger is better", and is a critical 
to the safety improvements afforded and other issues of vital interest to the public. 
[0085] Because of its unique design in which the control surface and the display 
co-exist, the invention can provide, at affordable cost, 5-10 times the effective display 
area (relative to conventional LCD screens used in vehicles today for navigational 
systems and other purposes ) and provide a substantially increased control surface and 
its attendant ease and flexibility of operation, while still meeting the FMVSS 101 
regulations regarding labelling of the controls. This means many advantages, can be 
provided the user. For example: 

[0086] Larger lettering or pictographs can be used, making comprehension with 
quick glances easier. For many drivers, this could be the difference between something 
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of great use, and something use-less. The lettering can be in ones language of choice 
as well. 

[0087] More data of the same size can be presented at once obviating the need 
for frustrating menu selection. 

[0088] The size of any tactile control devices such as knobs can be larger, also 
because they can provide multiple functions. 

[0089] Because the display is also a touch device, the much bigger nature of it 
also allows one to more easily select, operate, or " hit", in the case of a displayed button 
say, the desired function. 

[0090] Video and other data can be presented alongside text data. 
[0091] Video data and close-ups of certain sections can be provided. Another 
example is a navigational display of an overall geographic region with an inset near the 
instant location showing detail, both side by side 

[0092] Video and other data can be provided for driver confirmation or 
designation in the case of critical images and stereo camera pairs. 
[0093] One of the synergistic aspects is that at little or no added cost, the RTD 
can be provided in a size and configuration that can be used to safely and cost- 
effectively supplant many of the normal IP functions . This then lets the manufacturer 
standardize on the device for all customers of not only a particular model, but perhaps 
all models, with only cosmetic variations. This has huge savings implications, also in 
retooling costs. 

[0094] Because it resembles today's instrument panels, and can be used for the 
basic control functions of the vehicle, the invention provides not only a potential means 
of telematic connectivity while driving (e.g. with the internet, cellular telephonic sources 
or the like), but a much more useful display and control system capable of many more 
functions - including the primary vehicle control functions, if desired. I feel that an 
instrument panel incorporating the invention can be built at lower cost than a 
conventional instrument panel, especially as vehicles become ever more loaded up with 
navigational systems and other electronic functions incidental to the control of the 
vehicle. 
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[0095] In addition, it can enhance safety by promoting a degree of 
standardization between vehicles not now practical from a marketing point of view. This 
is because certain functions could be common, with the decorative and other portions 
customizable. 

[0096] Furthermore, the large display in the drivers line of sight without 
appreciable head movement, also provides a method to validate and confirm data from 
intelligent subsystems inside and outside the vehicle, and further allows the driver to 
assist such systems by designating areas of interest for tracking , stereo matching 
range detection and other purposes. 

[0097] Several embodiments of the invention are disclosed herein. The preferred 
embodiment utilizes a compact computer controlled rear projection display which on its 
screen (or an overlay thereon) is located a combination (any or all ) of the following 
elements: 

[0098] knobs switches, sliders, levers, or other physical control details as desired, 
similar to the Instrument panel of today; 

[0099] Tactile guides, either grooves or relief, or textured surface portions which 
may be easily and rapidly sensed by feel, but are and typically small enough in 
height/depth or slope so as to not appreciably disturb video images on the screen in the 
same location; 

[001 00] Programmable acoustic wave source(s) which can under computer control 
cause vibratory or pulse signals to be transmitted and felt by the user, whether touching 
or grasping a knob, or touching the screen. 

[001 01] In addition, the whole thing may include a touch screen on which arbitrary 
data can be projected and interacted with in arbitrary locations, as well as locations in 
which knobs, tactile guides and the like are provided. 

[001 02] The position or movement of any knobs (rotation) or levers (linear motion) 
or switches is monitored, as is the location of finger touch on the screen. In one 
preferred embodiment, the same machine vision system performs both functions, and is 
integrated with the computer control of the display and the force feedback. The total 
system is elegantly simple, and allows for a myriad of additional features. Particularly of 
interest are those in which the tactile aspects of the instrument panel can be tailored in 
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their entirety to the needs of individual users or the desires of individual vehicle model 
development. 

[001 03] To Recap some aspects of the above discussion, 
[00104] The invention seeks to maximize the available Display and Control surface 
"Real Estate", while greatly reducing glance time. Glance is reduced primarily because: 
[00105] Lettering is bigger and easier to see. Size and type can be varied as well; 
[00106] The control surface provided, and the Controls themselves are bigger to 
act on, requiring less fumbling, and providing more certainty of action on the "first try"; 
[001 07] Controls can be actively or passively felt, reducing or eliminating the need 
for taking ones eyes off the road, using, in the active case, force based signals. 
[00108] The lettering can be made larger because of two primary reasons. First, 
the invention, because of its placement of substantially all desired controls on the 
display screen, allows the display to be physically largest in the area available for same 
in the vehicle. And second, the controls, because they are reconfigure, allow the 
lettering for any given control function of the moment, to be maximized in size, and 
placed optimally, in location, and orientation for driver comprehension, even in different 
driving situations . There is little or no wasted space resulting from permanent lettering 
and spacing of control functions as exists today. 

[00109] And the display can tailor the coloring to suit the driver in certain 
instances. For example, I am red green color blind, as approximately 1/16 of males are. 
. For me the use of these colors in little LEDs on the dash is very confusing. Were they 
yellow and blue for example, I would have no trouble discerning their state. With this 
invention they can be lettering and indicators I can see well. 
[00110] The display is in turn large in the preferred embodiment, because it 
maximizes the limited space available and can share space between some types of 
control actuation functions and video displayed information. It uses in the preferred 
embodiment, a rear projection type display which can be cost effectively scaled to 
large areas of the instrument panel as desired. This allows one to entertain the idea of 
providing enough control functionality on the display, that a significant number of control 
and display functions can be incorporated, making the invention cost justifiable even on 
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economy vehicles. This too is radically different, as advanced features such as the 
invention provides have heretofore only been available to wealthy buyers of high line 
vehicles. The rear projection display has the added benefit too of providing a cost 
effective way to create not only a large tactile display surface, but an odd shaped and 
stylistically attractive one as well, also allowing maximal utilization of the available 
instrument panel space. 

[001 11] In the same vein, the RTD invention allows the controls to be located on 
the display, not requiring them to be arrayed around the periphery, in order to satisfy 
FMVSS 101 or other criteria. This in turn allows ideal placement of lettering and other 
instructions. And they can be displayed programmably in the center of a device, such as 
a knob to save more space. And visually displayed data can in some cases be "felt". 
And in some embodiments, the operator can be guided to location of an input or output 
on the screen by feel- either passive or active. 

[001 12] The controls can be larger, because once again the device is 
reconfigurable, so large controls can be used, as their function can be changed, and or 
augmented by other virtual controls which permanently occupy no physical space . And 
because of the display construction, space wasting prior designs incorporating LCD or 
other displays, in a manner to comply with the desirable tenants of FMVSS 101 , are not 
required. 

[001 1 3] The invention further discloses an embodiment utilizing an organic light 
emitting diode (OLED) display screen, which has a more compact layout than that of a 
rear projection version, while still having many useful features thereof . 
[001 14] Additional embodiments illustrate improved methods of sensing finger 
touch in order to provide "virtual" controls with little or no added cost, as well as physical 
reconfigurable controls - important issue in high volume applications. For automotive 
application, such sensing needs to be at once low cost, rugged, amenable to interior 
design and material considerations , and immune to common conditions experienced in 
the vehicle. 

[001 1 5] Further embodiments of the invention illustrate improved methods for 
providing data in knob and button or switch devices and interiors. 
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100116] Other Embodiments disclose improved sensing of button and knob 
positions, also in multiple axes and in the z direction into the control surface. 
[001 17] An improved version for the home is also disclosed, in which 
commensurate savings in switch gear and displays of individual appliances and other 
devices such as furnace thermostats and the like can be expected. In addition, the 
home application in conjunction with the car further reinforces the familiarity and 
commonality of controls and provides added economies of scale. . This application 
expands on the use in the home, particularly with a kitchen based control system. 
[00118] In one embodiment, the control and display device of the invention is 
located in the center stack, and judiciously can, if desired , replace substantially all 
controls of the vehicle even if conventional knobs and switches are employed, while still 
providing added functions and display screen space for advanced features. And it can 
do this at less cost with significant stylistic advantages. 
[001 1 9] Disclosed is a simple and easily understood format for the control 
surface/display screen is disclosed which allows what I feel is the preferred transition for 
the average motorist from the instrument panel of today, to a reconfigurable one of 
tomorrow. This is based on common Radio and heater controls, and their conventional 
position on the Instrument Panel of the last 50 years or more. 
[00120] Further embodiments include alternative methods of display and sensing 
on rear projection screens, including using scanned optical beams which can be 
advantageous for cost or signal to noise purposes. Such scanning may be 
advantageously provided by "MEMS" type devices, and can utilize the emerging 
availability of high power, "White" or colored solid state light sources such as LED's or 
Diode lasers which may be very desirable for high reliability in the vehicle. 
[00121] To sum up, the preferred embodiment has a rear projection display, knobs 
or other control details, tactile guides and programmable feedback. It typically has a 
large screen and control surface to aid vision and interaction, and is multifunction and 
reconfigurable. It is capable of displaying video, textural, or graphical images in a 
manner which can be acted on easily by the driver of the vehicle. The RTD invention 
not only has a number of quasi-conventional and other tactile characteristics that make 
it ideal for instrument panel usage In vehicles, it also may find application in other areas 



21 



such as aircraft and certain military control applications, control of heavy machinery, as 
well as in the home. Anywhere that for example, ease of use with familiar controls, 
large display areas, and ease of function in time sensitive situations are required or 
desirable. 

[001 22] Given the above it is therefore a goal of the invention to provide a safe, 
familiar, and easy to operate instrument panel which can reduce driver distraction and 
improve functionality especially for the elderly or visually challenged and for those 
drivers who are not technically astute. 

[00123] It is another goal of the invention to provide a display screen and control 
surface which optimally utilize the space available in the vehicle instrument panel and 
minimize wasted space required today to meet regulations or provide fixed lettering and 
devices. 

[001 24] It is also a goal to place the display screen and control surface where they 
can be optimally seen and interacted with, including the provision of controls which are 
easier to see and bigger to work with than many controls today. 
[00125] It is a further goal of the invention to improve safety by reducing glance 
time required to interact with a range of vehicle controls to well below the 2 sec 
minimum guidelines established. This is both with respect to the time to see the state of 
control, and the time to actuate and confirm same. 

[001 26] It is another goal of the invention to provide a display screen and control 
surface which have fixed, interchangeable, re-positionable, and programmable tactile 
references for ease of driver interaction by feel 

[00127] It is a still further goal of the invention to provide a means by which 
intelligent vehicle, transportation and highway data can be usefully presented to, and 
acted on by, a driver of a vehicle, and to provide a M man-in-the-loop M bridge between the 
vehicles of today, and the automated vehicles of tomorrow 
[001 28] It is also a goal to make the controls easy to use without an undue 
sequence of glances required, indeed in many cases with no subsequent glance at all 
beyond a first glance, and in some cases to allow control to be entirely operated by feel 
if desired. 
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(00129] It is another goal of the invention to provide a screen and control surface 
and associated computer processing for driver designation of points of interest in video 
images for stereo measurements and image tracking or analysis purposes 
[001 30] It is a further goal to improve utility of the instrument panel especially for 
those drivers with near field vision difficulty, physical difficulties such as hand 
movements or impaired finger function, and for those having difficulty with or aversion to 
technically sophisticated systems 

[00131] It is also a goal of the invention to provide in an economic and functional 
manner, large controls and easy to read data - much larger than possible today, and 
ideally suited for just quick glances and elderly drivers, and drivers in general who are 
far sighted and don't have close up glasses. In addition the focus of eye can be on 
objects outside the vehicle, as letters big enough to read even if out of focus. 
[001 32] It is another goal to provide a display screen up to 1 0 times bigger than 
conventional LCD type displays used in vehicles today for navigation and other 
purposes. 

[001 33] It is a further goal to provide a large enough display that side by side 
navigational or other data can be provided, with each data set fully readable 
[001 34] It is a still further goal related to size, to provide adequate space on the 
display to display in-vehicle road signs, whether telemetrically transmitted to the vehicle 
or taken with video cameras earned with the vehicle or in other vehicles and transmitted 
to the vehicle. 

[001 35] It is also a goal to provide space for easily seen still or video data to be 
presented which can provide the driver with much more knowledge of up coming road 
conditions (with the data for example taken from TV traffic cameras at key intersections, 
in narrow streets, in parking garages and the like, and transmitted to the car. 
[001 36] It is a goal of the invention to provide a means of virtual controls which can 
co-exist with other displayed data, when the control function is not needed, and to make 
these controls responsive to touch of the driver so that he or she can find the right 
location to make control moves, and or determine the state or move made, without 
looking with more than a passing glance at the control. 
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[001 37] It is further a goal to have a means by which a user of a virtual control can 
select the function and move along a line of action with a near unitary motion. 
[001 38] It is also a goal to provide a means by which the user can touch the 
control surface at a point and by which the force vector exerted can provide the control 
desired. 

[00139] It is a further goal of the invention to provide a reconfigurable instrument 
panel in a cost effective manner which can allow its use on lower priced vehicles as well 
as luxury cars. 

[00140] It is a still further goal of the invention to make possible the safe use of 
aesthetically pleasing programmable displays by a driver as the primary means of data 
input and output. And to provide these with large area screens and control surfaces 
maximally utilizing the available space in the vehicle. 

[00141] It is also a goal to provide means for changing these displays and inputs 
to suit the needs and desires of the user, and to provide means by which complex 
instructions for use can be presented in easily readable form co-located with the control 
function whose instruction is needed. 

[00142] It is another goal of the invention to provide tactile displays which allow a 
driver to "feel", in either a passive or active and programmable manner, the control 
location or information desired as well as see said information. 
[00143] It is also a goal to provide methods for sensing the position of knobs 
sliders switches, dials and other control details on a control surface using machine 



vision. 



[00144] It is a further goal to provide methods by which a TV camera system can 
discern touch location on a screen surface in xy and z, as well as vector of touch 
[00145] It is another goal of the invention to provide a touch type screen having 
controls which indicate to the user via physical touch sensible signals, such as acoustic 
waves, a condition, such as the control device touched, the particular selection made of 
choices concerning same (e.g. heat or wipers), or the magnitude of the setting desired 
of the choice made. This can also indicate what the current value of the choice or 
function is at the time of touch, not just changes therein. 
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[001 46] It is a goal of the invention to provide means by which multiple inputs and 
outputs (e.g. touch, voice and direct optical viewing of human or human actuated switch 
positions) can be used in concert. 

[00147] It is another goal of the invention to provide scenarios for prioritizing data 
and control functions to be presented to the driver in certain circumstances in order to 
aid safe driving and reduce distraction, stress, and other negative influences. 
[00148] It is a further goal of the invention to provide a large display and control 
surface which can be operative with hand held and other user devices which can be 
plugged into the control and displays system. 

[00149] It is a goal of the invention to provide means for both z (direction into the 
display screen) and force based input of single or multiple human or object inputs. 
[001 50] It is goal of the invention to provide a device, which may also include a 
touch screen, equipped with interchangeable front panels or overlays which have 
specialized and if desired, purpose specific physical devices for interaction with the 
sensing arrangement used. 

[00151] It is goal of the invention to provide a touch screen having a display and 
associated control function which moves around the point of touch, using for example a 
vector of touch, as opposed to requiring a person's touch to track a predetermined fixed 
display-a task requiring much more visual concentration (and associated glance time). 
[001 52] It is goal of the invention to provide a reduction in sources of anger or 
frustration caused by alternative voice based systems or systems having difficult to read 
and actuate controls or controls which are difficult to understand. 
[001 53] It is another goal of the invention to provide means for reading road signs 
and displaying same for the driver, and in some cases to automatically read and 
translate them. 

[001 54] It is goal of the invention to provide tactile control of the computer system 
used which can be customized not only by the user in general, but for specific purposes 
- what I call a "Mission specific" tactile and visual input and output. This makes it still 
safer by providing just the right tactile controls for the application at hand. 
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[001 55] It is goal of the invention to provide a mechanism for utilizing completely 
different and interchangeable tactile and graphic information, which can be in an 
interface which can in addition be widely distributed at low cost. 
[00156] Finally, a goal of the instant invention is to provide simple, tactile, means 
for people who are not computer literate, or partially disabled, to safely achieve the of 
enhanced vehicle control interaction and "telematic" activities. 

[00157] Further features and advantages of the present invention will be set forth in, 
or apparent from, the detailed description of preferred embodiments thereof which follows. 

Brief Description of Figures 

[001 58] Figure 1 illustrates an overall view of an embodiment of the invention 
located in this example in the center stack region of the instrument panel; 
[00159] Figure 2 illustrates further the embodiment, including detail concerning 
knobs and other controls, and examples of their use; 

[00160] Figure 3 illustrates further details concerning control details of the fig. 2 
embodiment; 

[00161] Figure 4 illustrates additional detail and features of the preferred 
embodiment additionally (or alternatively) incorporating a touch screen capability (which 
may also be in combination with the knob or other physical tactile selection or 
adjustment detail); 

[00162] Figure 5 illustrates a touch screen like fig 4, and having indented or 
raised portions of the screen or overlays thereon; 

[00163] Figure 6 illustrates alternative tactile screen designs with removable or 
adjustable large finger resting details and multiple "switch" or other control location 
details; 

[00164] Figure 7 illustrates acoustic or other mechanical wave generation capable 
of providing a programmable force based response to inputs, or control states. An 
optional sound based response can also or alternatively be provided, such as a 
loudspeaker sound; 

[00165] Figure 8 illustrates embodiments of the invention including 
interchangeable screens and overlays; 
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[001 66] Figure 9 illustrates a conventional LCD display based version of the 
invention equipped with an overlay containing tactile adjustment details of the invention 
and an optional touch screen; 

[001 67] Figure 1 0 illustrates several embodiments improving safety of vehicle 
operation using real time data made possible by the big screen area in which the 
invention enables- both to make it easier to see data, and to interact with it; 
[00168] Figure 1 1 illustrates a vector of touch aspect of the invention; 
[00169] Figure 12 Illustrates virtual displays with arbitrary start points on a touch 
screen of the invention or other touch screen; 

[00170] Figure 13 illustrates rear projection display embodiments of the invention; 
[00171] Figure 14 illustrates a camera based application of the invention for 
prevention of child deaths in vehicles; 

[00172] Figure 15 illustrates additional aspects of physical details and touch 
features of the RTD disclosed in previous applications; 

[00173] Figure 16 illustrates an embodiment of the invention employing knobs of 
fig 1 based on a traditional Radio layout, located in the center stack region of the 
instrument panel and its the reconfiguration under computer control into climate 
controls (also known as HVAC); 

[00174] Figure 17 illustrates the sensing and control strategies utilized in the 
above examples; 

[00175] Figure 18 illustrates a curved irregular screen/control panel embodiment 
further including venting; 

[00176] Figure 19 illustrates optical sensing of touch occurrence or position from 
quasi rigid body movement of the screen/control surface; 

[00177] Figure 20 illustrates an alternative display device employing a MEMs 
based flying spot scanner and laser or LED sources and the additional use of such 
scanners to perform sensing of both control details and fingers or other touching 
objects; 

[00178] Figure 21 illustrates embodiments for control of vision and projection 
processing steps, including methods for determining finger touch and physical detail 
location also in the presence of significant sunlight or other background radiation; 



27 



[00179] Figure 22 illustrates further machine vision processing and screen 
ZZ Figure 23 iiiustrates an alternate instrument pane, embodiment having a 

25] Fi9- * iliustra.es an embodiment o« the invention that utiles deflection 
onscreen material such as disclosed in 09/568,554 and USP 6008800; 

amera based physical detail sensing o, the invemion and, « des,red ^ 
1001831 Figure 26 illustrates an advantageous arrangement of the RTD ,n sloped 
instrument panel including power operated vents; 
1001841 Figure 27 illustrates alternative vent location embedments 
S3 i» 28 illustrates further TV camera based applicaUons also ,nc,ud,ng 
near IR light sources; 

poc: F RP F n aaB OBIMEMIS " c THF '^F-NTION 

'n .his application I will define the instrument pane, has having both a 
Ilay surJe and a lire, surface. In a conventional instrument pane,, the two are 

Tnler There is seldom a « between the two, except in certa.n veh,c,e touch 
« for navig*, and the BMW ,-drive ioystlc* relation to Ks screen (,n a 

Lupy the same physical area, wh,ch means that much more space .freed up 
both especially the display, relative to a conventional Instrumen pane _ ^ 
Wim While the ,ocat,on of the invent in the instrumen. pane, of the veh,c,e 
be anvwhl .he driver needs to interact, the basic embodiment of this appUcahon 

^nl^fren^atsanduU^esbo^nvenflona^obsandourer^ 
Und seiectfcn means, plus unigue .ouch screen ,*e capabM.es. As noted 
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displayed data. in the center stack 

rnn-iom Fiaure 1a illustrates a display and control surface, o. in 

knob 20 located within the display area. S.de vents 21 

[00191] WP > considerable utility, while prov.d.ng all 

certain sections with relief features acting as tactile references . And, as often wo 

1001931 . like 5 of fia1 a, but with a different aspect ratio. 

of a display and control surface 30. like 5 of fig la, 

This tactile version in this example has a knob 34 and six (for example) 



29 



™ it<if»lf (which serves therefore as a 

oonuo, surface,. The operatbn of these „»» ^- ^ 

curable virtual swtehes) - be more My ^ 

alte rna«ively he oufdenfs (bumps), and fhey may be a, » ^ sh P^s d 

triangular,efc,soas to bedis^^ 

screen any reasonable number o. knobs, swrtches. ^ ^ J ^ 

praoucable. all of which can be economically sensed .0 determme P 

sensing system of the invention. sufficiently 

[00195] 

tra „s P arent materia,, typically, like me screen ^f, a ndg«-» ^ 
surfaces to scatter lightfhere from. Such ^ * dWusive 

mad e by sandblasting or coating the surface ^ ^ example ,a 

smooth outer surface « des,red. Or the beads may a 
ma ,eria, described in fig 15 for example, and spaced away from me 

few millimeters or more if desired. nmipr t«j on their face and 

rnniQBl If the knob is transparent, the lettering can be projected on the. 

rot ational pos,t,on. Lettenng can *. * P I ^ ^ ^ off 

be of different sizes depending on the task at nana, o 

rrrrri s= 
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orft.nc.ion related date. And area between Knobs, or between «he Mb «. «-n 
edges is also available for projeded data. Some of this date oan reiate to tee knobs 
Jetton, and suoh a system complies wHh FMVSS 101 wbicb states that a control, such 
as a knob, must have date concerning «s funcfion on, or adjacent, said con roL 
,00198] Permanent lettering may also be used on the screen surface to denote 
knobs or indents or other control features. Generally however, the lettering >s 
llically Projected, in order teat I, can be programmably changed, as the function 

of the device is changed. ^ man * 
,00199, The use of one or more -classical" tectile physical selection or adjustmen 
Leans such as knobs, common to instrument panels today is a big advantege ,n that ,« 
is visually and tactilely closes, to tee instrument panel o.teday. yet often, the full 
probability -quired tomorrow and it is also thought to promote safer dnv,ng . 
ILo, What makes it novel and exceedingly usetol is that ,t has programmable 
visual and tactile aspecte app,oacbing, and in some cases exceeding, tee uti ty and 
capability of todays IP's, while at tee same time being reconfigurable ^ needed .0 
provide Led featuras and enhanced versafiltty. By doing so, safety ,s ,mp,oved, and 
user value is enhanced. 

,00201] Elements have been disclosed in the referenced co-pending appficabons. 
Ad di«onal deteils ara pravided in oteerfigures. While primarily hera ^ »«' 
preferable rear projecfion display, it is no, limKed thereto. The rear pro,ec.K,n ,s*ay 
Lgh. preferable for cost reasons (in larger displays a. least), and enables an ^ant 
solufion .0 tee sensing of detail and finger positions, and tee provision of tecfile feet 
100202] Typical maximum scraen/control surface sizes with the ,nven.,on when 
Lated in tee cen.ers.ack region are ,or example ina Minivan: by 
,2inches wide, and in a Luxury car, 13 inches high, by 10 inches Wide. Bote these 
esfima.es assume tee screen comes up ,0 neariy jh. top of tee instrument pane,, but 
not hiqh enough to obscure vision, j 

0 203] « a horizontelly extens^e anrangemen. is prided typical w,dte can to 16 
Lhes wide. Even wider screens /control surfaces can be used. bu. tee steenng wheel 

Stores can be placed. An example ofsuch an arrangement is shown ,n fig 2d and g. 
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„„,™.„t While most such screens/control surfaces are flat, they are not so 



the curved surface as we,,. be by 

mnnui It is noted that in one mode, the status or conn u 

Z ling o ly the, indent which applied to the control being wo*ed of ^ moment- 

illuminating omy illuminated with their 

e g. heat. After some time penod, all the .ndents cou ^ 

s^onir— , »*^^^JI^l toflBllB b.--- 
their positions would become learned, and the status 

at any time by touching the indent in question. 

SS The embodiment of flgure 2 senses positions o, screen mounted ^obs 

called "Centro, Devils" herein) mounted to the screen of the rear pro,ecbon ft* W « 
Tier piate in front of the screen toward the use, Sensing can be pert nmed by 
I Z of ectrfoal, magnetic, acoustic or other means fcnown in the art. but s 
ibiy illustrated here as electro- opflcaliy based, which has the advantage of 

2? "T^ZT, Ptobably the preferred embodiment, because*, 
T nd 2 contra, details are familiar to the driving public. Other aspects of the 
rflontrbep^edinadd^.buttheuser^.uld -^e- — 
exampie, always reverf to familiar control detail based operate of the necessary 

as a rear projection screen for a LCD. DMD, or other type of computer controlled 
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to0 .he rear proton system can incorporate nove, features to su,t the partrcuiar 

LrtotheLplrl.^e^sources.V^ means, or from sources 
122 within the vehicle itself, can be thus projected on the screen. 

I^nl^mputer 4, and approve controt -onses 
ToSlOl The new state of knob position after turning by the driver can be _ 
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state of being pushed m can be determmed by p ^ by 

screen as described in other en.bodi.ents sucb as^sT and ^ ^ ^ 

oa.era , 17 which can de.en.ine tha, two or ™» ^ has changed by 

kn own relationship and spacing . o, the apparent^ dimeter oj ^ 
, he change in magntfcation being cioser to the camera. 

be used to provide more resolution if desired. (o ^ g 

met a.,ic pin 106 siidrng ,n a me* anng for ^ 

™ * "* ^Tn^eTis ^ Iby 1 detent mechanism and the metallic 

allowing the center to be free. array 111 can be used to 

[00212 , While a specialized ligh, source such as LED an- ^ 
inmate the Knobs and other desired features on *es^re • ^ ^ 

aspects of the projector, one can choose the hght pro, ^ 
iiiuminateadis^^ 

unis0 n therewith) such as a marker -wh,ch * P |p ^ 

coior image obtained by camera 117 ypica y ^ ^ easierby 

toda y of the CMOS or CCD vanefy). »^ be „ acertain region ofthe . 

m e fact that its relative position ,s approxtmately know , 

screen. mechanical detents known in the art 

l00213] The knob rot*n can cpW* ^ ^ : . fee ,, g 
atdWerent positions in its rot*n <unc*o, and or ^ ^ ^ 

sensed dependent on rotational pos,on (o . . v«* » ^ 
alternatively provided by an acoustic source such as 125. ftp 
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rot a tto na, actors for a specific Knob in queston as deseed ,n USP 6,154.201 

Rosenburgetal, for example. ^^hx the user without 

roraui To provide excitation waves or pulses to be felt by the user wnno 
[002141 lopiuv „„„, ho er rf! en an acoust c source can 

resorting to electrical contact with transduce, o a ^ 

disDlav of navigational charts), space is sdveu 

Z be provided (aspect given the programmably recorrfigura le open*™ Vm 

tactile physical selection or adjustment means, can itseix q w a 

r:' g p'rog— acous* wave ^^^^TC-. 

setting of the Knoo oy or . |nstead 
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have been shown in co-pending applications. 

00218 Let us consider in more detail the operaUon of the inven«,on desonbed 
CAsshowninfnon.view^CthisOrnei.ius^ngdisp.ay screer '",n*a 
SL aspect ratio ) three Knobs. 115 and 116 and 119 are provided, preferably *, 
Tease of use. Whiie iiiustraled as veracaliy anayed, a horizon*! or any other 
p llg o, such Knobs or other lactiie seiection and adjustment means can b used. 

b e,ween heat, radio, and so fodh, and 1 16 for the amount (e.g. amount e*M^ 
„» is shown for seiection of vent locafen but ft is reaiized however, 
can be changed as programmed into computer 120 and .his change can be made 
Cste to me drivers actions, and in some cases to externa, triggers (such as a 
crisis or special information situation). 

I 19 1 Labelling loo is programmable. No. only are the fundus o, Knobs 
Til's and ,e like programmable and reconflgurabie (also in --ses p« 
ou. all function labels for a given knob for example can be displayed, or ,us. me value 

e« This .oo can be .ime dependent wfth the jus. selected label version only 
27« after selection is made, or some other ac«on is made which maKes ft. 
Tpl Priate Conversed -he folalfty of possible choices can be projected only as a result 

I other actto. In Ihe cased of Knobs, me label can be projected at approp na e 
por.Laround.heKnobasiscommon.oda,andor.heins1an.labe,can b e,n,he 

interior of the knob as disclosed herein. 

™ A driver can look a. soreen/confto, surface 105 and see the delayed 

rZ e. . .he worn "HEAT as shown. If he needs .o Keep his eyes on me rcad he 
"be ouches me Knob, .ceive a sound sensaflonvia me cars loudspeaKer . 
rylm124oraphysica.sensa«ondue to meacous«csourcesdc1 1 asv,br,or4ev 1 .ce 

sideways, or some other funcfen, me computer 120 can cause .he dnver to fee, 



36 



example a 100hz vibration via transducer 125 which indicate that the knob * a 
ZZ sLtorknoband fecurrar^te of p^ramrned operation- ,nth,scase 

K„„b pos« to n related cond^n for example, has changed, in another example the 00 

as to what the knob was for - i.e. having a sign with big lettering ngh. next to ,.. ). 
[00223 Such a status signal can option* be initiated through a vo,e command 
ZZ by microphone 126 and recognfced by sofiware such as IBM V,aVo,ce. ,n 

ST 12 — ,y, t hede fl n Sto no,mese tt ingcana,sobeac,uatedbyvoice. 
To check status of a display having al. ft. video data for example ob,a,ned from 

anay mounted in the screen105or near b y, picks up the voice, and compu. 10 
loesses it to determine that a status should be displayed, and subsequently, the 

computer voice saying HEAT as well or alternately. The ledenng can change ,n s.e 

or color when selected as well. 

'n any case as the knob is moved, the iettering projected « 
TaZ to indicate the new po-on. For example in one program as the knob115,s 
11 20 degrees rotation*, me fcncfion changes to wipers from heat an n so 
log the acltic source causes 2 hertz pulses, (or another choice of s,gnal ft. 

Z ll . AiternafiVeiy. or in addition to these pulses, an identifying sound, such 
a Tf miliar «. -y also be g enerated on command of computer 120 usmg for 
examie stored Wave sound files in me computer, which sounds may be broadcast thru 
the cars audio entertainment system to loudspeaker 124 or separately. 

™ Figure 2c iilus.ra.es a second posifion o, knob 1 15 in which a w,per 
[00225] g ed gs tnreewper 

„ ht mode and becomes the delay seiecfion knob. Aitemafiveiy. delay pos*ons « 
Ltcldtd In .he ratefiona, posifions ofknob 116 , and knob 110 can be for — ng 
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else- for example headlights. This would be of interest if the soreen in question was a 
high priority screen, in whfch ilghfs and wipers were displayed- two safety rtems of more 
importance than climate or audio for example. 

,00226] The display need not be only in the center stack. Figure 2d MM an 
arrangement (in this case having two knobs and 4 illuminated on demand indent 
switches (.o be described further below)in the minivan example of fig 1 us.ng al the 
instrument pane, space from a few inches to the left of the edge of steenng whee 1 50 
(around which it is dfficultto reach, and thus no. suftable for controls needed wh e 
driving ) to the farthest reach point to the right capable of driver in.eract.on Note that 
the rear projection display makes it possible to have a screen/control surface shaped 
, ike 1 55, where the region 160 is contoured to clear the drivers right leg. 
,002271 For illustration and comparison purposes a typical 7 inch d,agonal 
octangular display such as ,ha, used in the Jag X car (and one of the largest provided 

approximately 9:1 . This difference is one of the features fhat makes possible the 
presentation and action on many new types of data (such as video), possible and 
provides for dramatically improved visibility of al, dale and interaction .herewith us,ng 

r022 V 8 r° n The 3 knob based layout similar* that used in many vehicles today (and 
Ls familiar in this sense) where one knob is function, one knob is amount (magn„ude). 
and one knob a selector related to the function. 

,00229, „isno,edtha1whileoneknobsuchas115 can be fixed in function like 
Lbs in conventional vehicles today, for example always to act as a function selec, , or 
it can also be programmed to change tts nature once selected (i.e. onco hea, ,s 
selected, .he selection knob momentoriiy becomes a distribution knob for example, 
obviating the need forfhimknob 119 to perform Ms function). Alternatively, .« can be 
lays a selection con.rol , tut .he complete nature of whaf is bei^ selected changed. 
,002301 For example, fig 2e illustrates a table of values where all functions 
elec.edbyknobllSareintiresec^darycategory-i.e.notcrfticaltolhevehide 

operation. This screerVcon«ro. surface having this arrangement might only be ,n place 
when needed, and tire screen could for example revert to a critical category after 



38 



[00231 J r and knob 119 becomes the selector 

a v o,ume knob wHh a spectrum of analog se«,ngs , a n d kn b 

====== 

• i~~a k q diripr<; or evers or dials tor exaniyie. nu 
feel as well. Or the knobs can instead be sliders, or lever* 

STu-n se^o. o, adders can a,so be used a— onn a = 
r 0 0233] Foreaohse,ofohoioes,a te bleofvaluesl,ketablo2o anbecrea 

surfaces 180 and 181. each wKh a separate projector and camera system. Tb,s 
surfaoes 180 instrum enta1ion and conhols of the veh,o.e, s,noe all 

b ew inreaohofthednversnghlonefthand. WKh smal.er diameter or othe 
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W of the projector is preferably chosen to provide a signified depth of Wd. to 
allow an areas to be substantially in focus. This is made easier if the proved data ,s 
larae, so that fine focus is not required. 

[00236] The display software, in one simple example to provide speedometer 
readings, can be generated using LABVIEW by Nafiona, 
specialized instrument panel images can be created with the a,d of SPEED 
insttumentation design software provided by eGenuity Co. of Montreal. Canada. 
,002371 It should be noted that the fonction of a particular control such as a knob, 
can revert back to a standard state for example, after some elapsed time period, or 
upon a signal such as an alert from a sensor that this state should be resumed. Much 
the same applies to me whole screen/control surface, if its function has been 
momenta* changed- for example to present a video image of a critical accdent threat 
for example. When the threat is diminished, the normal control functions can be 
reinstated for example. Or they may be reinstated in order of some prionty. wrfh toe 
most cnttcal tooverall well being first, or those with the least time latency first. For 
example, headlights once on, likely can stay on with H. time considered associated 
2 them. But wipers may need to be turned on or changed in speed tetany q u,ckly. 
(unless automatic rain sensitive systems are provided for same, useful w,th th,s 
invention to all controls to focus on other matters). 
; 02 3 81 Clearty .here is a vast array of possible layouts of knobs sliders 
evers and so on, not to mention the ways in which different data can be displayed to 

in fig 3, and other types of controls which can be used alternatively or ,n add-on are 
then disclosed. 

,00239] A complete automotive control and information system can be but ■> 
ovelmannerashasbeendisclosedabove. In the form above ft represents ir .essence 
a programmable and reconfigurable approach to the conventual: mstremenP^e of 
today. And » can even be completely reconfigured physically, as discussed further 
below (see for example, figure 8 illustrating substttutton of screens or porfrons thereof,. 
[00240] However, the invention also comprehends the use of a touch screen 
capability, logically integrated wfih the features above, which can as disclosed operate 
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esired portion, may be reserved in certa.n opera** 

ch oioe o. selection or function, .he acoust* * P Me in the 

th evisua,displayoflettenng(and,anguagethereof.oo)areal p 

computer control unit such as 120. |ance 
,0024,, Thissystemhastheabilitytohave^re^n g ^ 

can aiso be used. For example when the knob* ^ 
annunciate its posrtion. Or the person ca ^ he wants 

^ c - ^Siliustratesfurtherdetailsofmesensingempioyedinthepreferred 
100244 , F,gure 3 ta«W £ ^ 200 illusttatlng one 

embodiment. For example^ 

method of sensing of knobs 201 and 2tv, M!L machine yiston software 

^ose image data is anaiyzed by software (w o(her ^ foatu res, 

,es«en« , oomputer 206 to determine the P-^ «J rejector, or a separate 

rrrdCro^re^ed.rough.soreen.andthen.^ 
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wit h respectto another ooiored baoKground) ■ Lafed on a .ember on 

tnro ughout. Ana—earrangernentcanhaveme marKer to 
.he oamera skfe of the soreen, and attached to a p,n suoh as 106. 
is turned when the knob is turned. 

O'clock, corresponding K*^^ 0 ***™^™^^ the camera 

^47, 'n°^ to « tem f^" e "^de tem ,,nen,a ril er 

sunlight behind the knob, the regkrn o, me rnarkers - w ^ 

chosen to be darker than its surroundings, ^^"J^ing system of 

computer 206, for example using the Matrox mil *.u 
in a Pentium 4 PC. 
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• r ,«<ao xA/hPre the Drojector illumination is 
roo2491 Another example, is shown in figure 3c where the pro] 

system, using projector illumination projected m an ann us 22 

X-.~«-^ b *^ , ^- , " ,,,,,,,0B " B 

[00250] d from human vlslon by 

" L vent ou«e, setUngs, if desired, with individual van. settings stored n the 
1 206 and whan the knob position determined, an appropriate lookup tabia 

wave source like 125 to provide p g f 0re xample.- And one 

^ m~~*^~J^^£ZU thltheothe, 
can ha va di^nt ^^^^ ^ actual , ocation of th e settings can 
so as to be distinguishable one from me o*e able posBion de ,en, 

be determined by the point at which the feel is ten. as p a 
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so to speak in this case the disp,ay may be varied as we,,.. The sensed indicator o, the 
innut to control the programmable wave source 125. 

m "or elp,e consider fig 3d iiiustrating a tabie o« vaiues o, dispiay an 
Sed back tor a Knob with 3 settings, with information to match, and de lator me. 
2 ^ration, the same knob having five settings. The displayed data - 

Tel some* ng change three umes in the one exampk, and five ,n the <* The 
Tnge can be fine same each posHion, ,ike a classica, detent, or can actuaiiy have 
Hn. vaiues. This wouid be ,ow medium and high fluency vibration for M ,3 

and ,h magnitude knob (in mis case 202, is used for voiume or stafion tuning The 

Lugh they couid be if Just a fixed set of stations was desired, one a. each detent 
loin (but with those varied to suK a seiected condition say, c,ass,ca, mus.c, vs. 

L But as you turn to the audio setting desired, the frequency ,s caused to maease, 
where as going counter-dockwise on me knob causes decrease in frequency. Each 

tack via a ioudspeaker if desired. Added knobs or sik.es can aiso be provided on the 
^ to selector adjust further fimcfions. Vidua, controis can aitemafively be 
^lhaJesc,b^be,ow.Thesehavemeadvantage j ofn^anentty 

ttlg up space on the screen, but are ,ess intuitive perhaps for *>me duve* , 
111 L eldedy who are accustomed to dassical physical knobs and skders and 

and software. 
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,0.257, SimMy as you go to turn the magnitude Knob. ft. programmable pulse 
Li Wen*" can, for example, be o, long durato .or bigb volume, and shod 

lb o sliding a s.ider or posing a swttch (ail physioa, tecttle dev,ces>, the same 
b r L or different choices o, vibrate, pu,ses or other physical wave s.gnals c n be 

ridge groove or indent in the screen as well, examples to be drscussed ,n figures beiow. 
^Lddverofac.rwpica.lyonlycontact.onesuchfunc.iona.a.imefbe.ng 

« Led to one handed function since fhe other hand is on the steenng wh ee l . 
Tp e can be Known ,o ft. driver to be associated wtth a Knob or a vrrtua, pos*o 

programmable wave funcfions could be rese^ed for cartings. For mp e 
iblns under 400 hz could be for Knobs, 400-6001* for s„ders. and ove 80 
touch portions of the screen. This isjustone example of many pern— of 
vibrational frequency, pulse length, pulse sequence and the l,Ke^ 
[00259] Figure 3e illustrates a secUona, view of a screen 255 where the add,„on 

screen and wtth pn.jec.ed light visible through .he slkter if desired. 

,00260] in this case an opaque mete, insert 253 has been placed , .he aeen 

Z more precise acfion ofa sliding metellic slidercontro, 256 wi.h a clear pas 

tbe slider can illuminate ft. region ,o be touched in addttion or attentat^ .o 
illuminating the region of the screen nearby. 

,002611 Figure3fillustre.esano*ertypeofswi.ch265, inserted ,nto .he screen 
266 this case o, a three position leve, acfion type, common to window lift cntrels 

268 is in the position shown in dotted lines. As it is pressed upware. martrer 266 moves 
ri-Z P for example Typically such a switch is held there until the dev,ce ,n 

1 this happens the use, releasesthe device and the martrer returns to «s ong,na, 
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«„ in „ In this case the original position as well as the new positions F up and P 
position. In this case u.« a v in w Pn tion Thus a complete 

up or down. The lever can have two positions for example 

selection locations. circumferential 

the second pos^n. ^ pos«o, the 

ft. Knob function can be changed. For and then after tha , 
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(whir on the screen proper, or a member such as an overlay attached thereto^ 
00266, in this manner ,hey may be used .0 sfcna, added data ,o the compute • 
UeciLy U sefu, in this case is a knob with mufiiple marKers or o~-^ - 
now shown. This aiiows determinate of knob position not oniy rotation*, bu, ,n 
directions transverse to the knob axis as well. , 77 „ nknob2 80 
,002671 AS shown in figure 3g. . the datums 275and 276 and 277 on no 280 

„ screen or other surface 2B1*» - camera 283 and computer 285 to ^ve from 
th e image of fhe datums on the knob, no, only the rofafiona, position of the knob b t 
aTtite transverse posfiion or movement herein as we,,. This a„ows added dafa to be 
provided to the system, as to any translation of the knob as well. 

00268, For example, a screen mounted joystick-like device of the rnvenfion can 

e provided in which a lever can be moved in any direCion. moving one or more targe. 
daL accordingly. This allows one to provide functions typical of a mirror adjuster or 

radio speaker balancing. 

100269 Another example is to employ a knob wtfich can be rotated to a des,e 
Uo . and then energized by hand by pushing side ways to select a fcncfioa And 
another desired input can be achieved by pushing vertically or in other d,n*tio , The 
function selected can always be me same, that is corresponding to the national 
position of .he knob, or « can be something also relating to in which directs or amount 
the knob is pushed in x or y from fis normal rest posifion. or a combmafion of bom 
,00270] in the example shown, this knob is used to adjust seats v,a up down left 

those as we.,. And it can be for speaker funcfions, selecting those ,n ngh, front left rear 
etc by pushing the knob in .he direction of 1 1 , f. 5, or 7 o'clock, where a compound x 
and /answer is of interest. In each case the display changes to the desrred taction, 
whe the knob is rotated to the function of interest to be selected. In each o these 
Uses fire longer the knob is pressed in the direction desired, the move tine, function ,s 
activated, similar to today's controls for al, of these functions on Chrysler cars for 
example. 
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,00271, This type of knob can be arranged in another fasttion, where selection ,s 
Z y soml ooTmeans .ban use o, Knob rofafion, and ft* Knob ration ,s used o 
ririven.n« m a g n«ude. ra «ber*anthe«n.theKnob te pressed,na gl ven^ 



direction. 



^ Figur e4,,,ustratesaddi fc naide.aiiandfea.uresof.heprevious 
t m o menfs here further incorporating a fouch screen Action (which may be used 

adjustment details described previously). 

S The .ouch screen function may be provided by means Known ,n .he art, 
uch a acoustic wave, resistive .capacttive and so forth such as .hose made 
Jiluch Corp, and Dynapro corp. Of parficuiar interest of fhe conven ^ 
Ln in *e art are piezoeiecfric or other stress or strain gage based systems such 
further described in figures below. 

,00274] HoweveritisaisopossiblefoseemefingertouchopbcaHy. intlus 
x ami one or more TV cameras are used, which generaiiy is the same camera such 
7TZ c sensing physica, details such as Knobs as disciosed above. Just as ,n 
1 le^fhe Z» 115 offlg 3, th. drivers finger or an object hetd by fhe dnver (or 

rrSec, displayed information on .he screen. ^tentatively, sources of „gb. , 

;„T 2 7 5 , Severe, methods of elecfro-opfically determining .be location of physica. 
touch are disclosed in this section: 

002761 • The firs, observes Ore users finger through ftk screen, wrth fhe 
finger substantially in contact with fhe screen, using light projected from the rear. ; , ,, 
(tvoicallv provided by the projector displaying images on the screen). ^ 
JEJ • Asecondmemodobsen,esfhefingeroftheu S era,somed,stence 

outward from the screen 



48 



[00278] • Another method observes the shadow of the users finger on the 
screen, generally while in contact with the screen, using light projected from the 
passenger compartment side. 

[002791 Another example of an optical method to determine finger presence and 
even depth of push by determination of screen distortion is also disclosed ,n the 
referenced co-pending applications and patents. 

,00280] Considerflgure 4a, wherein finger 300 of a user touches screen 302 whrch 
is sufficient* transparent and having, for example, a dKfusive front surface 303 facing 
the driver TV Camera 305 is located behind the screen and out of the way of the 
displayed image projected by projector device 310. Camera 305 is used to view a 
region of the screen 302 from the -ear. The camera image of features on or near the 
sin is processed in system computer 307, for example employing a SONY Pentium 
4 at 1 7 Ghz and standard image processing software such as MIL (Matrox imaging 
,ibrary) from Matrox, in Montreal, Canada. The camera may be for example a Sony 
progressive scan type with IEEE 1304 connection to image processing computer 307 , 
which may be the same computer as 120 used to sense knobs and the like, ,f desired. 
[00281] in a first case, the tip 301 of finger 300 in contact with the screen 302 ,s 
Lt illuminated through the screen using light 315 from tire projector 310 (or optionally 
by a separate source such as IR LED's 316 (dotted lines) located behind the screen, 
whose light, like that of the projector, passes through the screen from the rear. The 
reflected light 320 from the finger (primarily born the ti P 301 in contact with the screen), 
is sensed through the screen by the camera 305 as shown. The screen can be of 
ground glass or ground plastic, 3M Vikuity or other types, and may have drffusrve 
regions inside the screen material or on the front or rear face. A fresnel lens can be 
usedbeforethescreen(projectorside)ifdesired.Thissomewhatblurstheimage 

however, focus usually does no. have to be precise to detect finger presence at a grven 

™ Given the screen scattering properties required to make the projected 
display image visible to the driver and passenger, in variant positions, the useris flnger 
(or Angers) should be dose to, or in contact wflh the screen for best results. In some 
cases the flnger tends to fill in the screen fronl surface to reduce scattering effects 
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caused by surface roughness of the outside surface 303 of the screen (assuming the 
scattering is from the front surface, and not from scattering elements within the screen 
or elsewhere). 

[00283] For best results in the presence of bright projection images, it is generally 
desirable where possible to apply a vacuum deposited or other anti -reflection coating 
321 to the back of the screen to reduce backscatter reaching the camera from the 
projection source, (if the camera is operating using the projection light, and not 
separate IR sources for example). For the same reason, the camera 305 may be 
purposely located such that it is not at the angle of direct reflection off the screen from 
the source . 

[00284] If a separate quasi monochromatic source such as IR LED, or LED array, 
31 6 is used to illuminate the knobs, details, or fingers on the screen, a band pass filter 
at the source wavelength such as 322 (dotted lines) can be placed in front of camera 
305 in order to preferentially pass light from the LED source, and not from the projector 
(which projector could also include a band blocking filter at that wavelength- desirable in 
some cases anyway to limit heat reaching the screen . 

[00285] It is noted previously (and discussed further in fig 5 ) the touch screen can 
desirably have relief features such as indents or ridges , which may be for example 
shaped as a circular groove or as shown a linear ridge 324 in screen 302, in order to 
guide the users finger tip such as 301 to a certain location on the screen where for 
example the starting point of various command movements might be made. The indent 
or ridge can be shallow such that it can be felt, but not deep enough to cause excessive 
refractive gradients which would disturb an image say of a map or video display that 
might take up the whole screen surface in a navigational mode for example. Ridges 
and depressions in smooth surfaces of even 0.003 inches for example can be sensed 
(thickness of human hair), while even 0.020 inches deep or high can be used with little 
optical effect in many cases especially where slopes are small. . 
[00286] The relief feature may also be an area of different texture, for example a 
region rougher than the surrounding screen surface. This can be quite easily felt in 
many cases by persons (though not with gloves on - a drawback in some cases) 



50 



[00287] Where the screen is divergent via scattering microspheres (eg 3M Vicuity) 
on the rear surface thereof, a variance in texture on the front surface does not cause 
undue change in the projected image viewed. 

[00288] The image of finger tip 301 can be further, or alternatively, distingu.shed 
by other methods. For example the camera 305 may be a color TV camera, and if the 
light projected by projector is whrte in nature, the color of the imaged light from the 
finger tip will be flesh colored, and only flesh colored images can be looked for by .mage 
analysis software in computer 307. Since each driver's flesh may be different m color, 
one can teach the camera computer system during a set-up phase, by simply putting 
ones finger on a square or squares on which white (or another color) is projected. It ,s 
thus desired to match the return from one or more projected colors with an indicate of 
the persons finger. This simplistically could also be used to identify a driver for theft 
prevention purposes. Indeed certain aspects of the finger print can also be seen by the 
camera too -especially if the finger pressed in against the scattering surface. 
[00289] If the user wears gloves , the system can be retrained for the glove color, 
if required. The enlarged finger size can accordingly be accounted for as well as gloved 
finger contrast. Where near IR sources (eg 905 nm) are used, I have found good 
reflection from almost all gloves tried, black or colored. 

[00290] Another aspect of teaching is to for the camera computer system to "learn" 
the approximate size of the users finger contact on the screen in typical use, as 
determined by the camera system (by thresholding the image for example, and 
measuring the extent in width and/or height, or area, of the image at that threshold). 
This provides added value in that it can be used to roughly determine force. For 
example, the camera computer system can be taught a normal touch and a "push in- 
state of touch - just two highly differentiated states, which clearly will have a larger 
finger contact area in the "push in" state. This ability to designate a new state by 
pushing can be used to act as a selection method of a position reached by touch.ng. 
[00291] Other Z axis or force related sensing means-can alternatively be used to 
determine the location of finger touch or the movement/ force in Z which can be used 
with the knob states in addition. Such a suitable means can use the piezo-electnc force 
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queers located for example at the screen <*rners. such as 340 (those on other 
corners not shown for Canty) . See a.so for example figures 7 or 9. 
,00292] To illustrate, oonsider flg 4b which shows the ,mage obtamed by TV 
Camera 305 of three touch stafes-iight touch 320 indicate of 
broaching the touch condrtion and just touching the screen, the nam- touch toto 
where one wishes to indicate touch at a certain x-y point on the screen (,e. ,n the 
ST* the screen), and the pressing mode state 322 , typicaiiy used where one ma 

le i aiso be used for other cond«ions on occasion,. There can aiso be degree of 
;"rg,uchasnormaiandhard,,orexam P ieaswe, in one exampie. a compiete 

ranae of 5 analog pressing values could be obtained. 

Last Since the surtax of the screen 302 is typicaiiy scattering, objects that aren , 
: Intact with the sceen wiii be seen iess ciearty from behind - and can become 

aw ay from the camera. Thus a criteria for determining flnger presence on *e -aen - 
1 sharpness and coior, as we,, as degree o, Ugh, return (brightness, and almost 
^touches being wW ,in a strange on border of 10 m„,ime,es on the screen, 
but quite variant, also with the force of touch ,. 

,002941 To avoid having the camera and computer system exposed to ,mages 
which don't represent an intended touch type signal, it is possible to sense 
Lependemty that a touch condKon has occurred, for example w,th p,e Z oe ectnc 
ransdul located to define forces of the screen plate such as 340 in f,gure 4a 
^information is processed by computer 307 and then use this 
camera computer to analyze images on the screen. When a force or oarer touch 
"lion is detected the system is prcgrammed to look, minimizing the chance .fee 

transducers can also be used to excite the screen (and Angers touching same) for data 



detail 



52 



si-.—.- .-crrrrssr 

acoustic, optical or other. ^ (inner 300 with tip 301 in 

r 00 2961 Another point, apparent from fig 4a ,s that finger 300, w P 

consent in bright sun condKions for example. As can e p 

— •rrLT--™------- 
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respect to the screen and contnM surface 38 ^ fe 

the plane of the paper ^ 8areto ^ r ^^" Theproiedoral so il«nates 
seen by camera 385 located dose to the project™ mm. The projecto 
Tiers finger * 387 as we,., a„ow,n 9 Ks pos«ion and other vanables to be 

accordingly in computer 120 controlling the display. 
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100305, This arrangement has two ottter advantages as well. First, the pro.ec.or 
may be .ocated closer to the screen in the depth direction which may be helpful ,n 
certain packaging situations. Second the camera whteh is located off to the s,de, may 
be less affected by sunlight which may enter through the screen. 
,00306] This arrangement maximizes the light intensity as seen by the dnver and 
Lkes other artifices such as lender screens to spread the light in the horizontal 
(cross car, direction unnecessary (at leas, for the driver). However, as noted ,.may 
L desirable to spread me light vertically to suttthe driver, given .he relatively larger 
angles subtended by his vision in the vertical direction. 

[00307] Note too that by angiing the projector as shown, it becomes relatively 
easier to position an optional airbag such as 379 (dotted lines )behind the screen, such 
that it can blow thru the screen in the event of an impact . In this case the screen 
should have a tear strip or other means to allow it to give way and not hurt the 
passenger. While illustrated here in the center stack, such airbags are usuafiy 

sea, in this case . Or they are positioned in me steering wheei, where * 
a ,so go wi«h some packaging constraints. If new forms of con.ro, are used. Cher .ban 
steering wheels, this arrangement could more easily be in front of .he dnver. 
[00308, When using rear projection it is desirable tor best contrast to sh,eld the 
L where possible against off axis direct sunikjh. entering the display. Such sh,eld,ng 
can generally be judiciously achieved by the ptacemen. of the device in the instrument 
panel vis a vis the driver. Alternatively or in addition, techniques such as the 
rnicrolouvers described in Blanchard, USP 5543870 can be employed 
[00309, Finally it is also possibleto use back illumination of the users finger. In 
L case, also illustrated In figure 4a, the camera 305 sees toe da* or •shadow 
imag e of .he finger 300 touching me seen 302, wben back illuminated by light 31 
from the users side (in mis case light inside the car. This works Well dunhg day„gh, and 
in relatively well defined situations, but pooriy a. night unless auxiliary fighting . 
provided inside the vehk=le (e.g. from IR LED's wnich can be placed for example ,n .he 
Z of the vehicle behind the front seats, which are no, disturbing as they are ,nv,s,b,e, 
The camera is ideally used to see the finger touch location in x and y screen 
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coordinates when .he prelector source is sufficiency dimmed to eliminating background 
noise from the screen. The projector oan in some cases be switched off or its delayed 
ima ge darkened only momentarily, to avoid the impression of it being off, wh,ch can 
disturb the user. 

,00310] It is noted that the finger of the user does not necessanly have to be ,n 
contact with the screen, but can be some distance away, the distance being dependent 
on having sufficient lighting conditions which create a detectable shadow image. 
[00311] As in fig 3, The camera utilfced can be any commercial camera, capable 
of producing images which can be in.em.ga.ed as jus. discussed in order to see the 
flnger(s) of the person touching me screen, or something proportional thereto. And the 
same image when analyzed can contain information as to knob slider or other con.ro, 
such as discussed in figure 3 above 

,00312] As also noted in fig 3, best results may be obtained in some cond,t,ons of 
operation by using a CMOS or other type camera which can be addressed on a p,xel 
basis, to achieve faster operation. This will be discussed later as well 
,00313] While the discussion above has been concerned with finger or other 
Lges illuminated with light in the visible or near IR wavelengths, alternative* self 
generated radiafion of a finger or other portion of a body can be used, detected wfih IR 
TV (eg pyrc-electric) cameras 305 operating in longer wavelength regions. In this 
case it is desirable to use if possible a "Cold- light source in the projector, such as 
LEDS or light conducted to the projector by fiber optics from a remote light source 
,00314] It is also noted that one can preject light into the inner region of the knob, 
even if the rest of the screen is black, since the knob blocks the projection (if no. open ,n 
me middle) .. and one can in any case preferentially project more light in inner reg,on 
of knobs so one can more easily and rapidly discern their posffion with the camera. 
,003151 Note that one can also project more Infrared when something is 
eteoted, such as a finger or knob turning. The human doesn't see it, butthe sensor 



does. 
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Figure 5 

[00316] Let us now reconsider the screen of the above embodiments, wh.ch for 
example could employ a touch screen type just discussed, or other touch screen types 
known in the art where appropriate 

[00317] On a touch screen display, it is desirable to have tactile indicat.on of 
where to touch, ideally so one would not have to take ones eyes off the road or other 
activity (for example watching for enemy aircraft in a military sense, or watch.ng the 
stove in the kitchen in a home sense). This ability is provided in the invention in three 
ways; 

[00318] Relief details permanently on the screen surface which may be 
interchanged when the screen itself ( or an overlay thereon discussed further in fig 8) .s 
changed; 

[0031 9] A movable and/or removable relief finger guide; and 
[00320] programmable acoustic wave based force feedback sensations 
[00321] The use of such relief details was mentioned above relative to the above 
figures and is further elaborated on here. For example consider in fig 5a , relief deta.ls 
on screen 400 comprised for example of 6 indents (circular depressions) 401 to 406, 

and a further detail comprised of a thin horizontal ridge 407 sticking up 0.003 inches for 
example from the surface of screen 400. Alternatively the ridge could be a groove, 
and the indents, bumps. In addition the screen could be flat, raised or indented port.ons 
provided on an overlay member placed over the screen, either wholly covering rt, or 
only in part- for example on the drivers side. Typically the surface most outward toward 
the driver is made diffusive to scatter light to him and other passengers from th.s outer 
surface, but it is not necessary to do so. 

[00322] The indents can all be alike as shown, or if it is desired to differentiate 
them as it could be if they were to indicate numbers of a phone dial for example, the 
indents can be made of different shapes or sizes, or curvatures, depths, textures or 
other characteristics that impart a different "feel" (also programmable see fig 6). For 
illustration a further indent 408 has been shown to be square in shape to enable .t to 
be so distinguished, while indent 401 is shown to be shallower for an ind.cat.on. 
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opttonally be sloped at an angle vMh nt) and t0 better provide 

other applicat.ons, the screen/con transDarent l0C al protrusions or 

indentations such as 401 make findmg „ „ 

whi ,e still allowing image project on •» Qr depth , on me order 

of 0.020 inches or less for example <w ^ ^ gre Qn 

[00326, in Figure 5a, two optional Knobs a *» * ■ when 

— s zrzz~~~-r~ m ~~" 

l—I *-r- It", riruse^Itmovesbisttngeralong 
a nd ge line o, acton" suoh as 407. * • - ^ ^ as 

g. ridge (or alternatively a groove), and h,s finge P ^ 

^teC^ 

Cedwttbana,mos,oon,inuousmo«ona«ermese,eotton,amade. 
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iv= rtnne free form without the ridge or 
[00329] Such heat bar gestures oan also be done free 

groove for guidance, as is shown in figure 12. The 

1003301 

screen of fig 5a can also use indents 4U1-4 ^ 

lett ers projected in the indents to could change the 

42 3cana,so b eusedtotunetherad,o^ 

volume. Finger touch along R.dges 407, 418 and 

tone, as another example. am -404 are used for 

functions, and his finger went over to touch the in 

near indent 402 (which he had (earned where . was ^ ^ 

upward to the next indention farther up, 40^he ^ ^ - ^ 

controlled line of action 407. and so forth. He would no 

l003 32, Memauvel, two (or more, sets of ,nde ^ « pre ^ 

425-420 on the right side for example, for rad» selection. 

sig „a,s or Knob selectors can also or alternately be sed or *. 

l0 9 „333, h should also be noted that of the rrrten* 40^ V ^ 
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information Hea ^ . |ndicate how much heat 

was desired, for example. ^ as being for w|pers . 

( not shown) if this was projected by the compu ^ 

------ 

the screen 400. m ^ ^ ^ be 

such that when pressed it deflects locauy. v 

transmitted to the driver, indicative that his finger ,s located on t P 412. 
the latter prov.d.ng («f of relafvely sm v annular ring-knob 

====== 
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• This is a huge advantage where space 

imp acting fteconfiolfeaturesof *• '^"-^aslimportart examples, 
is lintted. as it is in automobiles and n«ary 

[00 338, AS illustrated in fig 5 above me prov,s,o of U c ^ 
Len used for automofive conffol acffvities allows hejser ,o ^ 
tou ch .he screen, as he searches fccfiie-iy by scannrng h. e o 

point or vaiue desired by ieaving his finger .here for we peno fl 

con.pu.er. Or he can use ofher means, one ^"^^ describ ed 

wh ich can be sensed, for exampia w«h .he p,ez 

elsewhere in .he dselosure, or by elecffo-opfical fl 9 a , opg ^ 

screen, thus aiso indicating ma. he has selecfed ft* funofm ^ 
d we„ or push in. For exampie siiding his F * 
b ar is dispiayed, .o indicate a different desired heat seftngs. 

touches the point this one of several P~**-« with ^ set Ung 

the screen, provk.es an indicafion of a state or #2, Then as he slides 

.presented at the point touohed (e.g. a heater biowen a sp e ^ 

accordingly to show the new setting (e.g. he moved h,s finger 

larger ridges or grooves may be used for tn ^ ^ 

interest for driving are large ndges acting a _ 

nlaces as well as 

say for example 0.2 inches high, which can serve as ^ ^ l^ ^ 

projected thereon, but in some cases ft. can be m g ^ 

Unas where the ridges are. or having the she f be a , s0 fig 

different looafions on the screen or removable out of the way 



6). 
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programmable nature of the dev,oe, provdes a future aMrty to 

gotten into a rental car, say. MM ^ 

zzz:r:^^ — 

computer to the image projector or other d.spiay 

===== 

and the instructors, and the controls easy to see. 

the conttol oomput ^ together ,„ one vlew . 

[00346] t Hoknnh 423 is switched to Temp as in Figure 5a. A 

* you. for example a en • « ^ Rom for exampte using . version of 

srrrs:^--— ----- 
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preferred version and aspect is: 



[00349] Glance; 
[00350] Touch; 



[00351] Move; instrument Panels- 

functions if desired, like wipers or heat, or whatever, ur 

discussed, or combinations thereof. movement, 

as disclosed in my spending «W» ** T 9 ^pa^ent of 

screen as touch, or detected in front of the screen, or ,n the passenger 

the vehicle. eQncin _ _ f commands provided by a 

l0 03 5 5, It is also noted that eJeCnX.pt.ca. oescnbed can be 

Uer touch or feature of physical «ac«le ° 

~rs======™ 

ter to the data projected and/or the force signals provided), 
computer to the data projeo tjme g 



63 



Thls is usefu, too when screens and ovenays are interchanged 

,003571 With the invention an elongate groove such as 410 ,n the screen or an 

«, „f Butter et al referenced above , but here is part of a programmable 
1 may be sensed without reguiring specials switches . and the groove of he 

al communicate back to the user by the force feedback mefhods noted. 

commonly used for "Seek" funcfions on car red,os o CD ^ ^ 
consider .rent view of a dKferent kind of greo e 4 ^ 
iiluminated by projected image from an ,mage pro,ector 
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m i„us posKions. The iUua.ina.ed indfcatton can be desirably ,n toe grt>ove and/ nex, 
,„ toe groove (shown). The operator cleany can fee. «ha. the reg,on to the left o, he 
narrowest portion 475 at the groove mMpoin. is a minus quantity, and toe reg,on to the 
right is a plus. Note that tois can even vary the rate of change, by having the system 
speed up movement as toe f.nger is detected cioser to the end points of finger trave, ,n 
the qroove, 480 and 481 respectively. 

,00361] It is noted that groove depth may also or alternatively be encoded to allow 
a -Fee," of position to be gained, for example toe depth of the groove at the end port. 
480 and 481 can be deeper than in the center (or conversely). 
T003621 The indication of being in the bottom or top position is detected by the 
LecUo optica, sensor of toe invention (e.g. TV Camera, and can be signalled by sound 
using the computer such as 120 to drive a loudspeaker with a high or low frequency 
Ld for example depending on which side of toe groove one is in. Or as tough, above 
a force pulse or vibration can be generated in toe scrcen to allow the persons finger to 
■fee,- toe indication, for example two states of pulse frequency corresponding to ether a 
plus or minus seek state. 

[00363] In a similar vein, a cross shaped groove such as 485 ,n screen 471 can be 
employed to allow a driver to select raar view mirror or seat position functions for 
example in two Cartesian axes x, and y, plus and minus torn center. _ 
.003641 Figure 5e is an alternative elongated relief member embodrmem, m <ta 
case a pair of ridges 488 and 489 ralher than a groove, woich illustoa.es other posi.K>n 
coding methods. Along toe edge are bumps such as 490-494which can be feltto 
« .he posttton o, onesfinger along toe relief member. Intois case toe bumpscan 
be coded if desired, as shown toe middle bump 492 has one bump, whrle bumps 490 
and 494 at the plus and minus extremes have two close spaced bumps each. 
[00365] As noted prevtously, a desirable aspect of such ridges , grooves and toe 
Ke. is that in linear form ftey can provkte a ■ line of achon", so to speax, whrchtoe 
driver can trace his finger along, in toe example from cold to hot, as one goes left o 
right The ridge or groove can be under or on fop. or surrounded by toe d,sp,ay as 
desired. 
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[00366] In fig Sf . a more sophisticated srtuation is illustrated wherein a plot of stock 
prices for the day has been displayed (in this cese in the same region as the previous 
heat control, as a stock price selection has been made), and the driver with a qu,ck 
glance can tactily slide his finger 4 9 5 along a horizontal tactile ridge or groove 4 9 6 to 
the point of the days trading (say 4pm near the right of the chart) that he wants more 
information on. Such lines of action" can be vertical as well as horizontal, and more 
than one can be on a screen (e.g. 4 9 7 in the vertical direction). And as disclosed 
above the line can be on a screen overlay which may be interchangeable, to 
accommodate different purposes . Today, such data is generally prohibited by law from 
being presented to the driver. But wrth the invention it may be possible to do th,s ,n a 
manner which can be safely accomplished. 

[00367] While illustrated relative to stock prices, the graphical information however 
displayed could be fuel economy for the trip, or tire pressure or other things relevant to 
the vehicle operation too. 

[00368] It is once again noted, that the switch or seek functions of figure 5 
preferably can be sensed with the same machine vision system used to see knobs and 
other functions on the screen, or the finger position at other points on the screen, erther 
free form on flat surfaces, or at positions with tactile relief features. And the finger can 
be on an overlay over the screen, which overlay can be more easily changed. 



Figure 6 

[00369] It is noted that the relief details utilized do not necessarily have to be on 
the screen or even an overlay thereon, but can be on a separate moveable member. 
Illustrated are alternative tactile screen designs with removable or adjustable large 
finger resting details and multiple "switch" or other^ control location details. Further 
illustrated is optional replacement of air vents with] display surfaces. 
[00370] Consider screen 500 for use in the tfinivan embodiment of fig 1 above, 
this display screen/control surface is provided with what ever knobs, tactile guides, and 
other features the customer or manufacturer has selected (realizing that such screens 
and appropriate software can even be sold in the after market). But in addition, rt ,s 
shown here with a guide 520, which can be separate from the screen (or an overlay 
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S - *e .op of *e instrument pane, can attend be in the screen and f 
edges, or eisewi customer for 

space To make this subs«u«on. may require in certain cars a change 

light in providing function of the .nvent.on, does not get in w y 
which can be vented through the screen 
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F. 9 ure 7a illustrates piezoetecmc or other mechanical wave generator, 
[003731 Figure ramus. v ^ abte to pu |se with varying 

mea „s such as a vibrator of .he type used ,n page* u« I * P ^ 
^sew^scapa^eo.p^ingap^— ^ ^ su* waves are in 
response to operator inputs, or changes ,n control states^ Typ y 

provided.suchasaloudspeaKe^^ 

[00374] The signal prov,ded can be a funcao P 

tactile adjustment or selects detail, e.g. vfe a Kn ob or ot e su* dev 

current or future states of vanous systems or settmgs. 

occurred, including random events such as an afcrm. ^ ^ 

l0 „ 375 , ^^' m ^zr^v^n^^ 

613. the touch is sensed by surtable means, fee e^by 

^rrn^^=rrr-.-^eby.brator 
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notengenderanomerrcsponsebymeuser.such as moving his fingertoanewlocaUon 

«?urh as P' on the screen/control surface. 

Asshowninsideviewof fig 7 b « should be noted that the acous,, wave 
can be generated in order to excite a detail connected to the screen, such as a knob, 
2 te red by pin 634 to the screen,con«ro, surface 613. and the force felt by ones 

'Z Iduenoy or another characterise o, the acoustic ex*«on to signa, to the user 

more the degree of pressing is determined using for example the method of figure 4 

suitable change in the projected image concerning this input as well. 

7Z Z Note .tat the sfcnals driving the transducers exeflng the screen (or tacble 

excitation discernable by the use, For exampie as one grabs a selector Knob and 
begins to turn K, the exdtation can indicate what posifion of the selector was ca For 

bursts alternatively, might signify the audio system. 

(00380, As shown in the front view, a bar 640 is projected on the scree 6 3. The 
pel moves his finger 610 aiong the bar , in this case from top to bottom ( O to H ). 
for example to increase the speed of a heater fan. As «he movement takes place the 
rLotricany driven exc«a«ton pulse rate changes fiom a tow o, 100hz, to a h,gh o, 
3 Ohz Thus by feeling me puise fluency, and when, if desired, possibly glancing at 
Tdisplay, the driver can determine the approximate fan speed (words tor which such 
as Low or Hi can also be displayed). \ „ . .„ ated , 

[003811 As noted the force generating transducer, in one example, can be located 

excite any finger in contact with it or a knob, for example, secured to rt). Or the 
Lducer, such as 620 can generate shear or bulk waves in the screen ,n order to 
effect the function. 
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[003821 As shown if fig 7c. the screen can be alternatively composed of a 
iaminate. for example 670 having thin plate members 675 end 676 and a liquid within 
680 (sealed around the periphery by seal member 685 and excited by piezoelectnc 
transducer 690 driven by high voltage supply 695 under control of the system control 
computer such as 630 to provide the desired affect in response to signals from the 
camera or other subsystem used to determine information entered such as by touching 
with finger 696 or knob turning or the like. Since the operator typically only touches 
one point at any one time, it is not generally necessary to selectively excite only a 
portion of the screen. 

[003831 Illustratively, a force feedback signal to the users finger in wh,ch an 
acoustic wave in the screen signals to the users finger that an event has occuned It 
can be pulsed, or of constant frequency, or varying repetition rate or frequency, or of 
varying amplitude and is an alternative to providing the user a beep tone for example to 
indicate that an action has registered. 

[003841 As an alternative to an acoustic source such as 125 physically connected 
to the screen to signal a leer of a variable and programmable nature to the user, a 
sound generator loudspeaker 130 shown in fig 2a can send waves 131 thru air against 
the whole screen 105 which can be sensed at the point touched . In both cases the feel 
is felt by only one hand, finger(s) of the user at any point on the screen, while the 
magnitude at different points varies due to the mechanics of the wave coupling and the 
manner in which the screen is constrained The frequency, pulse rate, or duration of 
acoustic information is however generally similar. 

[003851 It should be noted that the force feedback can be prioritized .for example, 
in one case it might be that the user didn't like the force feedback, and it could be turned 
off entirely. In this case the user could augment the visual presentation of data, usrng 
the conventional central details of the invention, together with relief based tactile cues, 
and if desired, voice to effect control functions. 

[00386] Force feedback may be used as a primary feedback mechanism, for those 
so inclined. It is contemplated that for such persons, the adaptive feedback of much of 
the vehicular operational data could be programmed into the system and learned by the 
operator Or the operator could chose his desired settings for each situation, and teach 
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.he system .hose. For example, for every touch of a knob or the screen some k,nd of 
ST- taught programmed force feedback would signal the user tha the , c^n 
SI point had been touched, and the state thereof. The user could achvate the 

can 1 .each the system in mis case (via keyboard entry using an optonal plug-, 

^general. Or I, might be that he would program .his only when hea. mode has been 
selected There are many possible permutations. 

m Note .ha. the benefits of tactile relief guktes for finger movement and e 
one to achieve a tactile guide even on a flat screen with no relief. Usmg 

nlnds within vertical limits in moving acress the screen on a virtual d,s P ,ayed 
rz albartorexample. In this case, in one mode, the computer is programmed 

sp^remmedtobe maximized . Tha, Is, when you fee, the screen 

maximally, you know you are on the projected bar (such as deleted ,n figure 1 a). 

you sZ on i« up or down however . the vibration ampMe is programme to be 

,o amplKude) of vibrate Is programmed to change, for ^ ' ' 

temoerature from low to high heat condition as you go from nght to left. By proving 

visual attention. 



Figure 8 



1003881 Figures illustrates embodiments of the invention including 
lbs and Cher «ac«e adiustment and selection details on the screen, or the dnvers 
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finger position at points on screens with flat surfaces, including those with tactile relief 
features. 

[00390] The screen alternatively can be a plain flat (or even curved) screen, 
covered by an overtay member containing the tactile details and features desired. Such 
an overlay is typically transparent and can be more easily changed than the screen 
itself in many instances. When the overtay is used, its outer surface (facing the dnver) 
is preferably diffuse, with little or no diffusive properties incorporated into the screen 
member in the region of the overlay. 

[00391] In one example the complete screen surface such as area 155 in figure 
2D can be interchanged to suit . In another example, perhaps only the region of the 
knobs would be changed. Or for example, a particular side region of the display could 
be overlaid with a replaceabla overlay. In this case ona might put the overtay ,n a 
region of the screen which had no relief detail of its own, putting any such detail or 
knobs on the overlay for example. 

[00392] Other ideas for interchangeable screens and overlays will become 
apparent in the following discussion. 

[00393] For example, one can have a rear projection based touch screen d.splay 
and control surface, equipped with interchangeable screens or cover plates overlaying 
the screen of the type just described which have specialized physical dev,ces for 
interaction with the sensing arrangement used. In this manner one can actually add, in 
an interchangeable manner, new screens or overtays thereon to your instrument panel 
which can be used for different purposes. This can enable one to do tasks by feel not 
otheiwise possible, or to tailor the instrumentation to suit individual drivers, or to 
provide different functions to suit different travel regimes or business purposes. The 
overtays can have printing on them, data projected on them, of variant material and so 

forth. , _r * 

[00394] Alternatively, one can optionally provide screens and cdntrol surfaces for 

completely different purposes all together , which may even be unrelated to the veh.cle 

and its transportation function. For example, in a military context, the screen and its 

tactile details may be easily changed, (plus all software relating the projection and 

sensing associated therewim) in order to provide a fire octroi screen for an anh-aircraft 
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Len is put bacK in piace of .he gun con,., scree. Some bas,c cnfica, veh,cie 

= = = = = = ==== = = 

window, which adds an additional optical element to the system, may 

interchanged. A group of levers could be on one sc eenjm 

wh or one screen could have markings on its tace ror emdi. 

screen, but in some cases an opaque area of the screen m.gn 
,003971 It is possible to have screens and control surfaces, where only P 
he screen or overlay thereon is changed as will also be descnbed. 

jet \.JL«^---^^--- nin * ,, ~r 

=========^ 

the car Or a screen related to a specific project, for use on a long top. <*" eral * 

, to screen that is interchanged, butthe software for generating and readmg the 
J! ditscontrolsaswe, The software may be permanent stored or ,npu«ed 
bv CD ROM or downloaded ftom external sources, to name a few possiMttes. * 

wbere one might indeed set-up a particular Humvee or other to^cal ve de fer 
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vehictes,orairsupport,orwnatever wnicns howsan 
,00399, TO inustrate ft. case o, an o «- ^ ^ 
options, removable (and interchangeable > W* « -nov g ^ 

====== 

(wh icb can in some cases be to see „ ther functi0 ns 

whatever Note that the same camera 801 can De 

T oC screen iocabons as disclosed above, assuming accent resotuboa 
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[0040 nj>ii«o ji« lie0 o,,rface with the screen 

read by the camera n Qf {hQ screen 

otherwise input data to the system. In th.s case the degr 

behind the card shou.d be sufficient* sma.l to al.ow accurate read.ng of the 

through the screen, ^ . , as just disC ussed for example in fig 5, or 
[00404] This overlay can be of ndges, as jum u 

====== 

corrected for slight errors in miss-posit.on.ng While it is aes. 

^cca ^devices sue, as dovetails or dowel pins In ^ZZ^ 

alignment correct required, computer element correouon can aUow 

be used to attach a temporary overlay for example. 

00406, Figure 8b illustrates another example in which a .he whole s^een and 

bs= ====«=. 

prov.ded. For example, on /hirh fl nume ric key pad of indent buttons was 

^ca.^orpho.0^ ^^J^,-*--, 

JToL example. ,—ly when no, needed tor phone communion, the 
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screen could setvetodisplay videoimages, and the con.ro. *dges and gtooves serve 
ni imoses related to vehicle systems. 

OoZ Thesoftwarecanins.rucmedriverrap.iywhat.unctionsmayb 
'""I anrith edHverieams qU ic W ybyfee, and sigh, where the points o, mferes, are 



provided, and the driver learns 
for interaction with the control system of the vehicle 

Fi gure 9 



alternatively be provided on the screen of the LCD display itself. 

£27 Consider in figure 9, an oveday member 901 having ridge or other sma, 

ZL 902 on U surface, is supported by 4 piezo - electdc based transducers. 
prot rus,on d . splay g2Q w|(h screen outer 

r^^r—JLthe^oationofpointoftouc^ 
by comparison offome signals using computer 915 (see for example, to**** 
USP 4 355 202 for an example of such an arrangement or Flowers e. al USP 

IZallowsadegreeoftouchpropodionanomeforceof.uchlobedetenm.nedby 

Zlg the transducer signals in computer 91 5 (eguipped with su,ta le stgna, 
Idl ing and data acguis^n devices no, shown,, allowing a seleCon f + ,ven 
l.Jnadebypressinginafva^ 

easy to achieve, and can be easly add* to ^ 

rrrrrrnrr^ 

push in of a knob, or ,he push of the screen with ones finger. 
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,004111 in add-on one can sense no. jus. a switching funcfion (pushed selected or 
no.) but a.so sense the amount of push exerted by the driver or other user of the 

lice. One such technique is to measure z force into the piane of me screen at the 
touched or gripped point, which then can be used give the desired input -what ever it 

oush into the screen was interpreted to the blower speed desired. 
S«l in another example, a TV camera may be used to Kentify that the dnve 
as touched a particular screen iocafion (e.g. where an indent exists, and,or projected 
data exists), and it is desired not ,ust to sense this state, but further and opfionally to 
sense the push into the screen at that point. 

0M13, ,nanop«iona,case,.he W nsducers i us«men, to nedarealsoabetobe 
Led (much as piezo electric sonar transducers can both send and rece.e, ,n wh.ch 

well, in the manner of fig 7. 

,00414] The ridge 902 (also noted in fig 5 and elsewhere), prov,des a tackle 
lion between the users finger and the overlay 901 and fiat pane, display 920 ,wh,ch 
loaiiy incorporates Mould costal materia,, a pair of poiarizers, su«ab,e dnve arcu-try 

s ,ope o heigh, of fire refief detail used such as a ridge or an inden.afion ,s small, so as 
no. to unduly distort, fire tight fieid of .he screen. However, ,arger reiief de.a„s can be 

; S 0 e 4 d 4 deS ' red O,her relief durations can be impiemented too as has been illustrated 
n fig 5 using groovesflndents or ridges/bumps of various widths dep.hs or spacings to 

Droaram control (as to their meaning).. 

SS Alternately me location of touch of a finger such ad 905 shown reefing 
against ridge 902, can be determined by any other suitable .ouch screen device 
Tiber such as a SAW type touch screen 

rand, model M14-SAW In M case .he touch screen would typically be ,ncorpora.ed 
into member 901. 
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borate such then it may no, be possible to u a so d^ay ^ 

Such a cutout overlay is shown in flg 9b. is because one 

[00418] 

ridge, indentation or other reUefrtetn. teke suc cessive readings and look 

for changes in location ( or even magnrtude t such date .s _ 
touch , and when change occurs, then begin tracing ^ ^ ^ ^ 

positional movements. conventional touch screen 935. A slot 

with a sliding contactor 931 . To v ne ( 

rather than the type shown in USP 5,982,352 w , 

-^"Trr;r.rr^r y ra— 
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example with spring 990 (in figure 9c) fe ^ 

rnna221 The knob or slider is always in the same place on in 

registered in computer 915 indicative of the Knob rotational posrt,on. 



2Sf Figure 10 illusUa.es several embodiments improving safety of vehicle 
mnAOKi The invention comprehends that preTerreu ei iuu 

SHI be made available, and because «he invendon maKes large s,zes prachca, 
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add^ccos^^re space .s^up as .any^ns^^vi^ 
touch properties which can tf desired exist everywhere on the control and screen 
surface) and not require space to be shared at all with the display. Indeed. The 

displays, such as for navigabon systems for example, having 5 or 7 ,nch ,agona 
lingular screens. In addibon the screen (which is shared with the controls, a d is 

s control surface i«se,f) can go right to edge o, the available instrument pane, area 
due to rear project and fact that controls are on screen and don't use surround.ng 

Z*n in addKion, «he screen/control surface can be odd shaped, to « around the 
rivers right knee for example. And i, can be curved or even indented in porbons * 
need^ largely due to me rear projection aspect. Again maximum uttobon resuKs. 
Room on the periphery may be for hard core control functus, or can b. made 
decorative or personalized using the ability of the projection device to d.splay ,mages 
which arbitrarily may be entered (if so equipped). 

[00428] In addibon, lettering can be placed in the normally unused porbon m 
Lie o, knobs since the lettering can stay tad in the horizontal plane, even as the 
knob is rotated. And I no, on the knob, the .ettering can be ideally spaced arouse 
k „ob. And ^ifo^ce^.^^^^^^ 
a„ow a «acble screen w*ich can also display images where the tactile rndrcahons are. 
,00429] Muchthesamea^umentsapplytotheconbols.whichcanbe bigger 
because tttey can in many oases be virtual, and take up no pemtanent room ** In 
other cases they can be physically located on the screen, share space w,«h ettenng etc 
And since data can be p roj ected into the middle o, knobs, the knob can be b,gger on the 
outside, since large lettering may not be needed around the periphery. 
,00430] Lettering is recommended in human driving ergonomrc stud.es to be at 

pa 0 p,e driving do no. have corrected near vision, as is the case ot ma y older dnvers 
1 use reading glasses. This is not easy to actbeve or any significant amount of 
Tattering if a display is sma, For exampfc. a 5 inch (approx, diagonal LCD d,sp,ay of 
the brand new 2003 Lincoln Navigator, is only 3 X 4inch approx 
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The lettering of the invenrion can be bigger es one might expect because 
1004311 . ™, r M 1, isn , all . The lettering can be made larger because the 
the screen is bigger. But war isn i a control function of the 

controls, being reconf,gurab,e, *w the ^ ^^ ^ orientation for 
reomen,. to be ma*m*ed in size, and placed further below) . 

dnve r comprehension, even in dWeren, driv,g the largest 

The display again because ft is reconfigurable. can How o ^ed P ^ 



[00432] Aspoim economically possible in the 

provide, by a substantial margm. the laigest s*e display 

— :r r^irr^rrraspectejioh.beiieve 

;rnlme ro isebesafe,yunde rt a k en.Someofmeseare.orexample. 

,00433, bigger and easier to see printing -^J*^ 
[00434] the ability to display two (or more) sets of data togetn t 

readable); relevant detail can be 

on ^ TV by wnteh one can safely and 

— ■ — 



system of the vehicle. 
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,00438] Features 2-3 ebove in .urn allow ene to provide a. low cost, usabte and 
easily viewable videos or video stills to the vehicle from roadskfe TV camera sources, 

Z. AndthisdatabecauseofKslarges*, can be reasonably and safely evaluated 

I^r" The abilKy of me invent to enable a much lareer display and associated 
Lrel surface enables many safety related possib«es. making the inventfon the 
logical stepping stone to the intelligent vehicles of tomorrow. For example at the 
Lent time there is a great deal of worK going on world wide relauve to adapbve 

factors issue is as dfficuK as the autom* control. The Invenbon ,s felt to prov.de 
needed link, by making the large display and control surface possible, and ,n turn 
priding the means by whfch the operator of the vehicle can perform two main 
Lnchons no. now available in an easy visual way. (nor really even diseased ,n the 
various technical papers I have found). These are, for example ; 
00440] f. To confirm data delivered by another source for action (or in action- 

i.e. a false alarm); 

.004411 2. To designate, on the data presented, an region for action. 
00442 Figure 10a is a block diagram of visual based control by the driver ,n he 
oase o both alert type informal and informatfon to be designated, displayed on the 

be the resuK of sensory data indicating a certain cond«on to be presented for action , or 
ran be at the drivers choice. 

Si For example, coming into any special place where added care , . needed 
ensore could trigger such a display. Clearly sensors onboard, such as malfunctions of 

such a trigger. Or stress indicators could also provide triggers, such as driver heart 

rate, or vocal titterings. „wm Q tir,n 
[0 0444] The invention with its big display can also provide a vanety of information 
notofherwise possible. One example is to feed raw video data tothescreen 
particularly video stills which would no. be too distracting if rapid motion occurred. 
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These can be from a vane* o, sources such as road sfcns (where he ,mage ca be 
Lgnified. to Mow one to see farther ahead- a good use in general of a specked TV 
Zera . Machine vision can aiso be used ,o .ad the sign and enuncate „ vocaiiy, or 
translate it if in a foreign language. 

Si The dispiay can aiso provide video feeds from cameras af inferseCons, ,n 
parking garages and in ofher vehicles. A„ wifh fhe purpose of giving .he driver v,sua, 
•nlal of interest, bu, wimou. reguiring sophisticated automated systems or speca, 
networks. Most of mis data can be transmitted using simple wireless means, for at 
"Z fOOfee. (such as .he X-10 device sold for $79 over the interne, .oday) and „ke,y 

ZST" IWeve the invention has considerable utility wfth respect to intelligent 
systems and may make it unnecessary to perfect some fully automatic systems to 
i ntelligent system benefits. ,n add-on, the invent- may be ,ega„y desrrable for 

[Ob^WTi*^' 68 For example Nissan motor company has filed US applications 09/963490 
Road ,ane marker .cognition by Funtshe et al, and 09/ 95f 499 Lane recogn. on 
apparatus for vehicle .Shirato et at, both of which utilize TV camera and machrne 
elology to he,p guide a car such that it stays wMtin lanes. The instant rnventton can 
2 such Ltty by allowing me driver to see Ceady me image(s) being processed, and 
accept them by designing mem as vatid to execute for a control he wrshes to 
automattcaay pursue. Or he can jus. have the machine visron system beep w^nhe 

mm, The beauty of .his is that tt a„ows one .o get systems on the road L wtthou, 
Ling me total of all known condWons sorted out. And . assuming sufficient vrdeo 

orage capa* in .he computer used for image analysis, you couid record rmage as 
Z in eel vehicles, developing an enormous data bank, also in con.unctron w,«h the 
drivers own view of the situation. 

0„Z ™s applies not jus, tt> lane markers, bu, .he tracking of tree lines. d,.ches. 
shoulders and other roadside hazards mat one does not want to veer into. 
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[00450] Its noted that one other function of the invention, as noted above, is to 
confirm an alert, for example the beep just mentioned indicating the automatic system 
has seen a dangerous condition. If one saw the image and it looked correct, then the 
automatic system could be making mistakes and you could turn it off. On the other 
hand you could confirm that it was correct, and if you wanted, even set it to a h.gher 
state of control. Only with manual backup of this sort, can perhaps such systems be 
made reliable. 

[00451] A big problem with automated systems is false alarms- 1 his is mentioned 
over and over in various world conferences. The human will not use a system which 
false alarms more than a tiny fraction of the time. 

[00452] But unless the automated system is totally perfect (unlikely in any first 
system), how then can it be eased into the motoring populace - which could save lives, 
time, and money? Such systems can be for automated cruise control, lane change, 
manuvers, and the like. 

[00453] Another issue re intelligent vehicles is range detection, and more 
generally 3D vision. This can be accomplished with stereo camera pairs but this is 
difficult to automate in the field due to the widely variant backgrounds and lighting and 
its effect on the image pair matching problem 

[00454] The invention solves this problem to a point, by allowing the driver to 
designate what he wants to track or see- greatly simplifying the match, as the computer 
need only concentrate in a given zone of the image, which has generally already been 
shown to have features which the driver could identify at least. 
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,004551 For example, fig 10b illustrates two cameras in the interior of the oar ,n the 
upper art o« the windshield pointing ahead. Camera 1050 is iooated in ft. example 
a prox mateiy above the driver so as to approximate his line o, sight, and « the master 

on me screen/oontro, surface 1051loca«ed on ins— panei 1053. The second 
c^era o, the stereo pair. 1056. is Seated on the right side of the » 
used to provide 3D information concerning what ever portion of the ,mage 1052 , 

eSla-ed by the driver by touching screen o, .he invention 1051 with his finger 1060. 
,00456] Alternatively in another instance the image on screen 1051 may be of a 

oad si n. and the driver may designate i, wifh his finger, in which case the sign ,s rea 
a, higher magnttlcation for example , and, if desired, translated info Engl.sh or a o her 

scire in computer 1065 connected to cameras 1050 and 1056. sue translated 
can be displayed in big letters or read verbally aloud by a text to speech program ,n 
Tmputer LI as provided by IBM Via Voice.. I. should be noted that i, there are no, 
enough pixels in the camera image 1052 to allow higher magnification read,ng of the 
sign from afar, that such magnification can be done optically, for example usrng a 
separate camera and lens system. The image can also or alternately be d,sp,ayed 
directly on the RTD screen as is. 

,00457, in another case a video camera can look sideways and acqu.re data from 
house numbers and store fronts along the noufo. The machine visfon system can hen 
identify them , and i, a particular store nameor house number has been ,nputt«M an 
say that the place in question wasjust about to be passed (or just was passed). The 
image can be displayed on the screen for confirmation, if des.red. 
,00458] Another example of video driving aids are the cameras 1070 and 071 ,n 
re ar view mirror housings 1075 and 1076 respectively. These serve to augmen the 
norma, rearview mirrons by providing a constant machine vision monitoring of bW vvr 
spots and meting lane traffic along the sides of the vehicle . Other cameras may pent 

their perfecfion in an automatic mode, the invention here seeks to use the machme 
visior 'capability primarily )ust to provkto an alert, displaying the image where trouble ,s 
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detected on the scen/confol surface 1051 for example, where the dnver can by 
Te tipper sylm of the invention that he wishes to tracK this image and report 

poss ible with the invent, with «s big display screen and cntro, surface. For 
^ZT • TVcamerasiniargetrucKscanshowtheroadahead.andlhisTV 



data be transmitted to cars behind whose vision is othen«ise blocked; 

1004621 • At intersections such video data can also be trans - 

^onstoplightsorpo.s.whichcannoUfyoncomingdr.ersofstopiightcond^ 

STT A1sha^.es,videocan,erascan1ransm rt .ovehic,esabou tt o 
enter the cuive the road conditions they will encounter. 

Ill • Video dala can be provided in or around a towed trailer And the 

[00466] This can also allow belter backing up. if a camera ,s on the back of the 

traiipr to view the road behind. 

• Afni g h1.da te canbe Pro videdhomac«veinfraredc,meras,for 

Lple using .he same cameras as iust men«oned but with an auxiliary near IR 
M ' Camera data can be transmmed to .he vehicle from inside parking 
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and "seeing with your own eyes". Other examples where one would wish to oonfinr , or 
or view information are: 



[00470] 
[00471] 
[00472] 
[00473] 

[00474] 

[00475] 

numbers or the 

[00476] 

[00477] 

[00478] 

[00479] 

[00480] 
[00481] 



[00482] 
[00483] 
[00484] 
[00485] 
[00486] 
[00487] 
[00488] 
[00489] 



Baby in rear facing child seat crying; 
Children in minivan third seat fighting, one screaming; 
Display triggered by data from outside; 
Video feed of a crossing; 
Video feed from a vehicle ahead; 

Camera data from a camera viewing roadside markers, house 



like; 



Stop light or other, 

Video feed from sensor outside car- lane etc; 
Kids running in front of cars; 
Lanes passing etc; 

Sensor signals such as suspect range data, radar etc; 
Sensory alert from car sensor- image or info stored . note image 



could be a cartoon or other representation, rather than actual v,deo; 



Time of stress.- special screens activated 
Rain snow darkness fog; 
Traction charts and info; 
Night or fog sensor and video; 
Range data plus image; 
Persons in vehicle - kids baby; 
Medical emergency; 
llllI4Hai . vehicle malfunction emergency. 

00490 Finally the big screen is useful as well to display images taken w,th 

1 , near R ser or LED sources and appropriately bandpass filtered cameras ( to 

^Iseelrough^forexa.^e.Lc.ger.wave.Rootainedwi.hpy^eo.noTV 
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cameras can also be dismayed, as an attemative to a heads up display such as used by 

22 ^ in addttton mere are many P ro,eCs areund me globe aimed a, sensory 

a e .he aforementioned Nissan assigned Patents on TV camera and mach,ne v s, on 

logy to helpguideacar such that it stays w«hin lanes. Some of this technoiogy 
'o! ver very difficult to perfect in a «u„ automatic mode. I Know o, no reference 

screen tactile control and display device of the type disclosed herein. 
;r 492] Furthermore, the ,aroe display largely in the dhvers line of s,h, w, ho 
appreclble head movement. a,so provides a method to validate and ^ 
n H«ent subsystems insKle and oulside the vehicle, and further allows the nver to 
alt such systems by dealing areas of interest for fracKing , stereo match.ng 
range detection and other purposes. 

t 004931 The invention may operate using a powerful general purpose 
Lrocomputer such as an Intel Pentium 4 , equipped with image processing software. 
Z aTge era, purpose processing is involved, at different points in time me processor 

screen as a computer and Keyboard or other device when the veh,cle ,s stopp* for 
example, butthe camera and sensory processing capability can be „me shared 
2 reguited for the control and display of norma, instrument pane, funcftons, wh,ch at 
most times require very little processing power, video camera applications such as 
Those above as we,, as other .asKs can be aconom,ca„y automated in some measure as 



a result. 



JS^ 1 Figure 11 „,ustrates vir.ua, displays of controls wttharbftw star, points 
on atouch screen of me invention orother touch screen capable of the required 
response to human inputs needed. 
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,00495, in one embodiment the computer inlets the display projector to cause 
nino.reterenceofad.playedobiectto-iump-to^pointtouct.ed^r.rs, 

point becomes a reference for further action. alhar1 im 0 n 
00496] For exampie in fig 11a. consider.be projection of a thermal bar 1101 on 

point of touch. As shown the starting point is a. 67 degrees, .he current cab, 

Z ell. If the person had firs, touched the screen a, point „. the, point would 

Z represented 67 degrees and the bar calibrafion would have been re-s, e 

Z rdi gly As *e person moves his finger from side to side from th,s pant, the 

nltr 1 05. moves accordingly to indicate the new(84 degrees) seteng desrred - for 

iZelopointmOwitharightwam finger movement in the hotter d,recfio n. A 

1 J L 1 123 can provide a convenient fine o, action to move aiong.we 
I497, in otherwomsthe display is commanded ,0 commence from * present 

rihe poin, you touch. Whatever point you firs, touch „n a manner ,nd,ca„ve o . 

displayed bar. Knob or fire like. This poin, can be one w„h an rnden, or o*er reKe, 
teaLI which your finger finds by fee,, or « can be on a plane surface of he screen. 
It yl push anywhere, having firs, commanded ,ha, -hear is variable a rssue. One 
I^rmandfiTvadabieis.o.ouclrwi.h.hefingerabigd^^ 
screen. Or ,0 s*y fire worn "hear ,0 a voice recognition system, or Cher me,hods 

d a, a firs, poin, desired, which ,hen signals ,he compuler ,0 prov«e a d,sp,ay on 
he I n of a Ly knob, «er or whatever. The display is preferably onented so 
h Uhe s^ g po Jor frame of reference of fis movemen, is established a, fire po,^ 0, 

pushing in is ,0 be considered fire fire, point of reference in Ms context. 
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===== 



Ian convent .ouch screens requinng ,he toucn «o tracK a prede.erm.ned *« ; . 



display 
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Figure 12 

[00505] Figure 12 Illustrates the use of touch vector inputs on a touch screen of 
the invention, usable with either rear projection or other displays such as flat panel 
displays. 

[00506] As shown in fig 12a it is also possible in a similar manner to that described 
in fig 4 and 5 above to determine a vector of touch from the shape of the image of the 
finger in contact with the surface of the screen. Consider for example that image 1 225 
of the finger contact region on the screen 1232, is made by the driver of the veh.cle 
touching the screen in his normal way. However, the image 1226 is that obtained when 
he "pushes" the screen at the point of touch in the direction away from the driver (to the 
right in North America). Image 1227 on the other hand is that when he - pulls" toward 
himself. This pull in this case, may mean less than the usual force, not easy to 
register, and in some cases difficult to tell from the "normal" case . - 
[00507] Such "push and pull" is however, aided by having a relief detail on the 
screen (or overlay) such as a groove or ridge. Consider finger 1230 exerting push or 
pull sideways (right-left) force on groove type indentation 1231 in screen 1232. 
Because of the approach angle ALPHA to the screen, and the groove shape, the finger 
will not appear the same in its TV camera image for each condition - normal, push and 
pull These characteristics can again be taught to the camera- computer system. 
Typically the finger zone becomes oblique and weighted in the direction of touch one 
wishes to indicate. 

[00508] As another more pronounced example, shown in figure 12b consider 
image 1245 of finger tip 1246 pushing on the protruding relief ridge 1250, (as well as 
the screen 1248) versus image 1251 representing pulling from the other side of the 
ridge The two image conditions are much different and easy to learn and detect, due to 
the presence of the finger on one side or another, and the blocking of the finger by the 
ridge which takes place. 

[00509] A finger vector of touch can be used for control purposes. And the screen 
display moves around a substantially fixed initial touch point controlled by a users vector 
of touch (rather than move to different points on a fixed display to correspond to where 
the touched point is moved by the person touching). This aspect is radically different 
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t™ i = m aware of and is achieved herein by having 
than any known touch screen operatron I am aware of, ana 

a dispiay responsive to the vector of touch, which aiiows an urg,ng of the touched pent 
in a chosen direction to command the display on the screen. 
,005101 Forexample conskferfig 12c where a screen 1260 ,s touched - port P 
C1 1262 The display is controlled fo put a virtually displayed *er 1255 ngh on 
he Z r po f P indica ve of lis posilfcn along a sliding control path (bar 1266). At a 
I u «1 he display looks like M of fig 12d after pushing with ones finger a g,ven 
al n, o «me wL is rela.ed to a posftfve (push, or negative (pull) increment ,n 

position along the path, and a corresponding increase ,n a vanable, ,n tm case healer 
bit sp J. The previous posKion o, the heater con.ro, s,ider is shown ,n dotted 

ImH l« should be noted Iha, by pushing harder in one vector direction or 
anoL (and causing a corresponding change in me finger stature delected), the 

sply (and i,s corresponding funcfion conlrolled) can be caused «o move more rap,dly 
^ a d sired new slate. Thus bolh direcfion and rate are controllable ,n th,s manner. 
Isl Notemafmepie^eclnclransducerbasedforceorslrainsensrng 

1m of fig 7 or 0, by comparison of the fransducer signals, can aKernafrvely or ,n 

Ion since me sktewalis of fhe rtdge or groove allow a force «o be exerted n that 

I lways push or pull generally exists on fhe screen when the r W ge or groove , n 
ZIZ IL, H— relief delails can be used, but are much eas,r 1o push 

as 195 on the screen as noted previously in figure 2f. 

If should be noted Iha, me veoter louch can provide date in more « an one 
Z: F orexlmp,e,consider fl gure12ewhichshowsandge1280onascreen1281,n 
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,op view For a finger 1285 in posMon W I. function when «te finger urges the ndge 
H 0 h oMeft as delbed above )cou.d be heater fan speed. Whereas, a. pos,.,on B 

Tomer posittons aiong the ridge) can be defeated by variations ,n shape or otter 
Tnsasdiscussedhenein. Attemattvelyor in addition, programmable wave 
Generators can be used for this purpose. 

[00M61 Other arrangements can be done tor ai. the other funcUons des,red as 
r o " 517 l It shouid be noted that the line of action provided by a ridge or groove 

control function (e.g. hea, , can be selected byjusUouch.ng that barjher* - 
separate motion needed. On the other hand, if there is a selechon button or no . that 
sed to select, in which case the bar does no. have to be present un.,1 seated. 
J! , should be noted that in bofh^g 11 and 12 it may be **-^« 

1 , or some other re,ief feature in ttte semen for ones finger to locate ,n. Th,s can 
LldTIIdrivingifKisdesi.b^at.hedisplayof data and other information be 

de.ec.ion of said change indicating a touch in the locabon ,n queshon. One example 

2 materia* which change color due to heat. Such materia, on a screen at 70 
lis F when touched by a human a, 98 degrees, can change coior reasonab y fas, 
slTaUhe came, of «,e invention (assuming » is a color came.) can detect, 

Sf The.aresevera^esofdisplayswhichcanbeusedbymeinvenUon. 
including flat panel displays such as LCD, Plasma, etc. However!; it is ,e„ tha rear 

alntages in a cos. effecttve manner. They a,.ow .he bfcges, size, >n curved or 
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desire, and providing an eiegan, and styiistic solution, without a to. of speoia, tooiing 



,00521] An example of present art in rear projection (albeit, using three CRT s 
Lther fhan LCD or DMD ohip based projectfon) is USP 6185038 by Yamaguch, a. a, , 
entitled 'Rear p roi eotion soreen with light diffusing sheet and projector us,ng same , 
assigned to Matsushita corp. , 
,00522, A typical rear projectbn display iayou. of the invention is shown ,n more 
etail in flgure 13. In this one example (of many potential layouts) the projector unff 

the ashtray The screen assembly 1307 generally comprises 1-3 components The M 
lellesltotheprojeconlens 1320., an optfona, « ,n tha 
form of a flat Fresne, lens 1324 that collects the expanding proton l,gh tarn he 
projection lens and provides a uniform brightness of moduiated «*^**™* 
mllaled by the liquid crystal or DMD module1330) acress .he area of the screen. An 
uter element 1 335 of the screen assembiy is formed by a rear projection screen w ch 
may Pe formed o, any suffable type of rear projection screen materia,. For exampfo, the 
screen may be formed of a sheet of glass or transparent acrylic having embedded 
therein or coated on a surface thereof a plurality of diffusing parades, such as glass 
beads or micro-spheres, as is well Known in the art. The function of the screen ,s to 
effectivelydiffusetheincominglmagesoastoenableviewingoftheimagefrom 

dffforent viewing angles. Alternatively the screen surface can be roughened ,n order to 

scatter light (e.g. like a ground glass) such as region 1337. 

■00523] Athirdelementofthescreenwhichmaybeontheoutside.or.ns.deas 

shown 1340 , is a lenticular lens array or a holographic grating capable of preferentially 
directing light in a certain direction, typically honzontally. This is optional in .he 
automobile case, and desirable ', a hofographic graflng ,345 wffh.be property o 

recflng lighl preforenflally .0 .he driver and passenger locations, bom hon.on.ally and 
Tats er degree veracafly. Such a device could ideally account for the limffed range of 

dffferen. from .he home .heatre applicatton where lenticular screen type grahngs are 
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typically used, where the position of the users eyes can be anywhere in a large 
horizontal region. 

[00524] The grating could alternatively also be constructed to have a zero order 
diffraction component to send light between the seats to back seat passengers 
[00525] The projected image is directed at the screen 1 335 by one or more 
mirrors, such as 1350 . In most vehicles, the largest screen in the center stack location 
would be 16 inches high and 9 to 10 inches wide - almost the same aspect ratio of 
HDTV sets, but with the long axis in the vertical direction. 

[00526] The distance from the drivers eyes to the center of the screen at the top is 
about 30 inches, which distance however, varies by driver and their seat location . 
[00527] In the example shown, the display is projected directly on the screen with 
no angular shift to accommodate the drivers position off the center axis of the car. 
However, I have found that it is desirable in some cases to shift the axis of projection to 
make it more along the drivers line of sight to the screen and control surface . 
[00528] Note that one can "tile" the projection if desired, for example using two 
small projectors one above the other (in this instance), each having a display region 8 
inches high, and 9 inch wide say to fill, between them, a center stack screen 16 inches 
high and 9 inches wide. Each unit is accordingly less deep in the Z direction, but may 
there may result a linear demarcation between the two projection fields which cant be 
used, (which might be acceptably covered by a chrome trim strip say for stylistic 
purposes). If this is done, it can be helpful to reserve the top portion say for video and 
touch screen data, while the bottom is at least largely reserved for knobs and other 
details. One can even tile the displays to point at different angles, to maximize light 
sent to the driver for example. One can also "tile" the camera systems or other means 
used to see the knobs and touch points , having for example one for each projection 
unit. 

[00529] While invention has been shown with flat screens, it can be provided on . ; 
curved screens as well. An example of such rear projection screen for use in vehicles 
(however without the control features of the invention herein) is depicted in a recent 
patent application by Hirose, et al of Nissan Motor Co. referenced above. The instant 
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invention is compatible with that arrangement , and can have matching curved shaped 
tactile devices on its face such as sliders, ridges, grooves and so forth. 
[00530] Figure 1 3b illustrates a curved screen version of the invention on which 
images are projected by projector 1354, in which a knob 1355 is inset into the screen 
1356 such that it rotates on a flat surface 1357. Such an inset 1360 can be made with a 
milling cutter for example, assuming sufficient thickness of the material of screen 1356, 
and the arrangement can work even if the screen is curved in two planes. 
[00531] A knob can also be inset at an angle Theta to the surface normal, such as 
knob 1366. However this makes it more difficult to project data through the middle of 
the knob from projector 1354 without suffering obscuration of the projected image data. 
[00532] When using a slider such as 1 368, it is generally desirable if possible that 
it move in the plane of least curvature, in this case out of the plane of the paper. 
However a slider of the invention can track around a curved surface as well. And too, a 
tactile ridge or groove can generally follow the curvature of the surface without difficulty 
in use. 

[00533] The knobs and sliders are detected ideally using the camera system of the 
invention as previously. 

[00534] Figure 13c illustrates an alternative projector unit 1300 which can be 
comprised of a DMD micro-mirror device 1370 having a motor driven rotating color filter 
wheel 1372 as known in the art, to sequentially filter projected light 1371 using 3 color 
filters (typically red, green and blue). However, we can also use this filter wheel, with an 
added IR filter 1375, to once per cycle project invisible wavelengths (typically near IR, 
such as 0.8micron) on screen 1373 which can be seen by the camera system 1374 of 
the invention and not the passengers. All sensing if desired could be done during this 
cycle, which could occur 30 times per second say ( for an 1800 rpm filter rotation, one 
IR filter such as 1375 , per wheel). This allows the projection of any light pattern 
desired for sensing on to the screen, while still achieving rapid enough measurement. 
[00535] Alternative to use of the camera 1 374 to obtain control or touch data, one 
can use the DMD chip itself into a scanning system, obviating the need for a separate 
camera. In this case a separate detector 1380 would view the screen through the DMD 
field by use of beam splitter 1382 . In this case during a measurement cycle, every 
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1/30 sec say, the DMD is programmed to scan a flying spot of light , raster fashion or 
other wise, whose return from the screen region is detected by detector 1380 and 
analyzed by computer 1 385 to determine locations of knobs or finger tips or other 
control items discussed above. Because the knobs sliders and other physical details are 
in fixed positions (for the screen in question), and the finger touch is generally or 
exclusively (depending on the particular design) in the regions of the tactile relief details, 
it is not necessary to raster or other wise sweep the interrogation beam (e.g. a spot) 
over the whole screen region, but rather only in those areas where data is expected. In 
addition, once data has been determined to be in the process of inputting, it is generally 
only necessary in the next few seconds to scan the region from which it is detected, 
since the driver can generally only use one or two fingers of one hand at once. This 
same logic for scanning may be applied to the camera based versions above in 
scanning their image fields. 

[00536] Some miscellaneous thoughts on aspects of the driver interaction relative 
to the invention are: 

[00537] - The driver typically grabs a knob or the handle of a slider with two 
fingers (and sometimes just pushes the slider with one); 

[00538] - The driver typically can actuate a touch screen displayed switch 
function by "hitting" a indicated zone with his/her forefinger (typically) in a forceful touch 
type motion (especially in times of stress). The driver can react to more complex video 
data in the same way, if it is clear what to "hit", for example a displayed picture of an 
object of interest; 



[00539] - The driver using the invention can slide a forefinger on the screen 
to move a virtual lever, and the invention provides unique tactile guides for this; 
[00540] - The driver can twist a virtual knob on the screen to rotate a virtual 
knob; 

[00541] - The driver should spend the least possible time in looking to find 

the control, or to determine what state it is in, or what he did with it; 

[00542] - In touching functions when trying to actuate something either 



tactilely or virtually, the control needs to be of a certain minimum size, to match the 
persons finger size, and be easily visible to the driver . This matching size can be 
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altered using the invention, or left at "one size fits all". Virtual controls can be altered 
programmably. Physical ones require interchange of some or all of the display/control 
surface; 

[00543] - A tactile feature of the invention comprising a groove or ridge (or 
other indentation or protrusion) on the screen should also be matched, but cannot be 
too deep or sloped that it provides confusion in the video image displayed in the region 
where it is (unless it is planned not to present detailed data there); 
[00544] - It is not likely, but not impossible that the driver might use more 
than two fingers in working the device. 

[00545] In the simplest case, the invention is likely to be operated not as a 
general touch screen but only as a specific location one- the points where the tactile 
relief features are. 

[00546] And it may be operated in a manner that may be dependent on the other 
controls settings. For example only is shown an illuminated touch point "A", if a value 
of variable "B" is selected or a value of "C" appears. Clearly there are many 
possibilities 

[00547] On a given physical screen and control surface of the invention, the driver 
knows, or can soon learn where the physical features are . This isn't just the knobs 
and other physically protruding control details, but also the shallow indents bumps 
ridges and grooves. Note person can soon learn location of grooves relative to control 
data. 

[00548] If the user always uses just these (at least doing one particular regime of 
operation of the system), then from a sensing point of view, this simplifies image 
processing considerably The camera can preferentially be windowed so as to only look 
in the known regions when this regime is operative. This is a major processing 
difference, and it may also be simplified further if the types of driver inputs looked for 
are limited, (one example would be an instrument panel which was limited in the regime 
in question to responding to Knob turning and finger line of action movements in certain 
regions). 
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[00549] One arrangement of controls is to have a knob closest to driver be a 
selector of real time functions. Lights wipers, downloads, communication, video to hit 
etc. The secondary functions are in a further away knob such as climate, audio etc. 
[00550] Before considering further details of the function of the invention, three 
illustrative examples of the utility and synergy of the big screen (by automotive 
instrument panel standards) and tactile features enabled by the invention are now 
presented. 

[00551] One of the most exciting examples of application is to an economy car, 
which often must economically appeal to two sets of customers with much different 
outlooks and needs: 

[00552] Retired persons on a budget, whose first concern is safety and ease and 
familiarity of normal operation, and who may be reluctant to try new things 

[00553] College kids, who are technically astute and given to try new things. 

[00554] Historically economy cars have been simple, with minimal equipment, 
which means little if any added functions were available to the driver. However, the 
invention can provide, in a single affordable system, vastly different features and 
benefits to these two classes of drivers (and everyone else in between). And the 
invention can provide new intelligent safety features for both. 
[00555] Elderly persons will appreciate the large displays and easy to operate 
knobs, while the young will appreciate the vast ability of a completely reconfigurable 
dash to access technical and telematic features. It should be noted that not only would 
the software and displayed digital data and Video be different, it is very likely the two 
cars would have different tactile layouts as well- made easy by the interchangeable 
screen and control surface aspects of the invention herein. ) s 
[00556] In a family travel example of the invention, let us consider the screen of 
fig 2 similar to what could be installed in a 2002 Chrysler Minivan. The family is driving 
down the road and is able to access via telematic means lists of motels in the vicinity to 
include large pictures which can be displayed on the screen, able to be seen by the 
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driver and other passengers. Alt matlvely, the screen can display pictures of the kids in 
the third seat, or a baby in a rear facing child seat, taken using low cost cameras within 
the vehicle. When inclement weather approaches, the screen provides a display of 
infrared able to see thru fog or at night using an infrared camera located in the front of 
the vehicle, a feature now available on some vehicles, but requiring an extra "heads up" 
display. Again, the large display, typically located high in the center and center left 
portion of the dash, allows the driver to see easily. Other display locations can also be 
employed. 

[00557] The same vehicle as either example above for example, can also be fitted 
with the invention with a layout aimed at safe telephone communication and project 
activity while driving, two concerns of the business user. The dash of this car, can at 
virtually no extra cost, be laid out quite differently than say a vehicle aimed , temporarily 
at least, at serving a family vacation function, (since the layout can be changed via the 
invention, the vehicle can be one and the same) 

[00558] For example, for the business user, the rapid access to lower level vehicle 
functions such as seat and mirror adjustment, or even climate control and entertainment 
system is of relatively small importance, compared to the ability to easily use the 
controls related to telematics and business project activity (for example relating to tasks 
involved in formulating a proposal capable of winning a big order). 
[00559] The reconfigurability of interchangeable and fixed screen aspects of the 
instrument panel can provide specialized computer generated and mechanically 
different screens for young or old, those with bad eyesight or hand coordination, 
(accessibility), those with difficulty understanding technological functions, and the like. 
Red-green Color blind drivers can tailor their instrumentation to suit as well. For 
example, Blue and yellow are much easier to see than the common red and green used 
historically to signify certain conditions. 

[00560] Totally unique is the fact that the physical layout of the dash control 
surface can be changed as well as the computer represented "Computer screens" of the 
display, with the two aspects synergistically in concert. In addition, some changed 
portions can be in effect, sub screens, which can be created at home, purchased in 
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stores, delivered in cereal boxes and a host of other things, not necessarily solely 
limited to vehicle control. 

[00561] While shown with one projector in rear projection embodiments above , a 
number of projectors for example in fig 3 may be used in a "tiled" fashion, much as one 
sees "video walls" today. Each projector displays on a portion of the total screen area. 
This also can be advantageous in that it subdivides the area into different regions - 
some maybe more important than others, perhaps having more reliability, or resolution, 
or some other feature. One can also consider having screens of the invention used in 
concert with sections of conventional controls. 

[00562] It is noted that more than one camera can be used to see the control 
details and other information such as finger touch locations on the screen, in order to 
provide more resolution. 

[00563] It should be noted that displayed data next to (or even within same, if the 
device is big enough) a knob or other tactile physical selection or adjustment means 
usually at a minimum displays the selection or adjustment choices - e.g. wipers, heat, 
air, cruise; or lo, med, hi for example. Alternatively or in addition, pictographs, colors or 
patterns may be used for example. 

[00564] In addition other information can also be displayed, such as more detail 
about what the choice means. This can even include effectively displaying the 
instruction manual for that function if desired. 

[00565] The displayed data possibilities above do not represent the totality of 
novel functions which may be provided, but are ones that I feel are of considerable 
utility. They may also have: 

[00566] • An ability to switch automatically to a new safety oriented state on 

input of data from sensors which indicate such is called for. This state can be activated 

manually as well; j 

[00567] • an ability to switch among several screens; | 

[00568] • an ability to be switched from one to another,; as well has have 

individual functions triggered from a variety of human command inputs, including touch 

related activities of various sorts. 
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Stylistic issues 

[00569] The screen is typically clear high optical quality plastic (E.g. Lexan, 
Plexiglas) with a diffusive outer surface. The screen material may be darkened if 
desired, to make the system innocuous and or invisible when not turned on. This is also 
achieved by having myriads of diffusing beads, which make vision into the device 
impractical. 3M Vikuiti black beaded screen material also is specially designed to be 
dark for contrast enhancement purposes, and fulfills this goal as well. This material 
exists as a film with embedded beads which can be laminated onto most screen 
materials. For example in some cases it may be desired that the screen be extra 
rugged, for example of laminated bullet proof plastic in a military vehicle. Or one might 
alternatively chose a softer material with more "give" in case of accident . It can for 
example be of colored plastic such as black, which while attenuating projector light in 
rear projection (and thus requiring more lamp power, also reduces sunlight entering the 
optical system and prevents visual sensing of the innards of the device. 
[00570] The screen can be surrounded with a bezel, such as brushed aluminum or 
wood, which can also be removable to attach interchanged screen faces. An overlay 
on the screen can also be so composed. 

[00571] Generally speaking excessively variant form or texture on the screen 
surface is not desirable for either display or touch purposes. However, if used, the 
location of significant texture details can be sensed when the system is in a set up state, 
and stored in memory so that these details do not interfere with data to be sensed. 
[00572] The invention has been described herein primarily for automotive 
instrumentation use. However some of the embodiments are useful for home 
automation, military vehicles and fire control systems, and video gaming. Everywhere 
that intuitive and reconfigurable controls are needed with little learning curve. 
Reconfiguration ability may be desirable where sp$ce limits having numerous different 

i 

devices too. j 

[00573] I should note that the whole device might be removable from a vehicle, 
and able to be taken inside the home, where the projector system could become a 
home TV . If one changed projection lens focal length, and removed the screen, one 
could project on the wall of ones home! And external input devices such as keyboards 
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can be plugged into the system if desired, to allow the display and computer to function 
conventionally when the vehicle is stopped. 

[00574] Also noted is that the display of the device, could display information from 
PDAs or lap top computers or cell phones , if same were plugged in . Particularly of 
interest are small devices for example Palm pilot PDA , 159, and cell phones, whose 
normal controls are hard to use. They may be connected into the system of the 
invention , for example through a USB port or firewire (IEEE 1 394) port of computer 
120, perhaps using a connector plug right into the screen/control surface of the RTD 
itself such as receptacle 158 in figure 2D. With suitable software for the RTD control 
panel in question, one could then employ the various knobs, buttons and virtual displays 
of the invention for use in activities connected with the plugged in device or devices, for 
example searching a contact list on a PDA using a knob of the RTD. The RTD has an 
advantage here too, like the CD example mentioned elsewhere, that information 
concerning the connection of something electrical can be displayed right next to the 
connection if desired, helping understanding. Or it can be displayed in Big letters say, 
somewhere else. One region of the screen for example, could be reserved for such 
information, allowing one to form a habit of looking there for same. 
[00575] While machine vision and electro-optical techniques have been shown to 
determine control detail and/or finger position , other non contact techniques , while 
less preferable may be used such as inductive or capacitive devices . It is possible to 
consider sensing methods contacting the screen as well, but the elegance and 
interchangeability of the invention is diminished. 

[00576] It also should be noted that while automotive applications have been 
highlighted, because of their need for tactile function, the invention is usable for a 
multitude of such as point of sale devices, to Information kiosks, to home automation 
and control functions, to factory controls, etc. 

[00577] The screen layouts of the invention above do not represent the totality of 
novel functions which may be provided, but are ones that I feel are of considerable 
utility. Generically, these novel screens generally have one or more of these common 
threads: 
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[00578] • An ability to switch automatically to a new safety oriented state on 
input of data from sensors which indicate such is called for. This state can be activated 
manually as well' 

[00579] • An ability to switch among several screens, both computer 
generated virtual displays, and inter changeable tactile screens; 
[00580] • An ability to switch (line of site, larger print), to higher clarity for key 
data and/or elimination of non critical information. This also be a function of the 
individual safety or other needs of different drivers; 

[00581] • An ability, related to the above, to reduce distraction via a choice of 
distraction reducing screen which provides minimal vehicle information, but maximum 
assistance with distracting influences 

[00582] • An ability to be switched from one to another, as well has have 
individual functions triggered from a variety of human command inputs, including touch 
related activities of various sorts, plus hand motions, head motions, finger motions, and 
the like. 

[00583] In operating the invention or other novel instrument panels, there are 
various ways to signal the system (for example to change the function of the screen or a 
portion thereof), aside from the actual turning of a knob say, controlling or selecting the 
function in question. For example, voice recognition to determine a desired action 
from the drivers spoken word or words. For example, One can say "Heat" and after the 
voice recognition program such as IBM Via Voice recognizes the word(s), the heat 
screen related portions can be displayed. This could be a complete heat related screen, 
even including graphs of temperatures, temperatures in and outside the car and so 
forth, or it could just be the heat portion of a screen which would continue to display 
other information. 

[00584] Such signalling can also be done using proximity / range detectors, such 
as shown in USP 5,362,970; 5,880,459; 5,877,49jl; 5,734,172; 5,670,787 and others 
by the inventor and his colleagues, to determine the position of a persons hand or 
finger which he can wave or point at some aspect of the device which would cue the 
function desired . 
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[00585] It should be noted that the sensing of a persons finger position or motion 
gesture thereof in the air in front of the display or other instrument panel portion is 
made easier, as one typically knows where to look. (e.g. near a particular knob). And 
the illumination is also controlled in the sense that the projector of the display, maybe 
used to favourably illuminate the region as well. 

[00586] The invention allows the driver to signal the control system in some 
additional ways which can be quite useful. For example, using the invention's unique 
ability to determine finger gestures from screen indications. 

[00587] Let us further consider the situation of using the system to sense locations 
and gestures where the fingers are either in contact with the screen or very near it . This 
is the sensing situation illustrated in fig 4 above, but in this case with several other 
examples of finger combinations or movement. 
[00588] Some choices are: 

[00589] • Number of fingers touching the screen can signify an answer: one 
or two fingers typically, usually right thumb and forefinger; 

[00590] • Cartesian position of one finger on screen, typically right forefinger; 
[00591] • Difference in position of two fingers (spacing)- typically thumb and 
forefinger; 

[00592] • Angular position of axis line between contact points of two fingers 
(knob twist / rotate gesture)- typically thumb and forefinger. See description of function 
in fig 11; 

[00593] • Angular pointing direction of one finger. 

[00594] These commands are quite powerful, and several are innately intuitive. 

[00595] They are sensed by sensing finger position or change in position of one or 

more fingers, using any of the means disclosed, and where applicable, analysing with a 

computer sequential readings in order to determine the result desired 

[00596] For example, One finger hit could be on, two is off. 

[00597] Alternatively one can touch the screen with two fingers if you hit the left 

one twice, or leave it on longer, you signify something different than doing the same 

with the right one, and so on. Or you can just hit the right side of the screen with one 

finger, or the left side to signify something 
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[00598] It is further noted that the methods of detecting touch disclosed above are 
able to detect the touch of multiple points at once- a major advantage over many 
conventional touch screen devices. This allows unique advantages. For example 
functions to be touched can be interconnected in their operation. It is possible to with 
one finger touch a point on a card of the invention signifying a person, while with the 
thumb or other finger touch a icon signifying a telephone (thus commanding the 
computer of the car to call that person). While this could be done in sequence, it is 
often easier to just rest ones fingers simultaneously. 

[00599] Note that the screen surface, or an overlay thereon, can be changed to 
create different indented or raised styles to suit different user needs. These could be 
specified when ordering from the factory, or dealer installed, or even user 
interchangeable They can also incorporate customized printed on writing or designs, or 
contains specially added features, such as more knobs, or bigger knobs or keys (for 
elderly drivers, say). Similarly, the projected images could be bigger too. 
[00600] It should be noted that displayed data next to (or even within same, if 
room exists ) a knob or other tactile physical selection or adjustment means usually at a 
minimum displays the selection or adjustment choices - e.g. wipers, heat, air, cruise; or 
lo, med, hi for example. Alternatively or in addition, pictographs, colors or patterns may 
be used for example. 

[00601] In addition other information can also be displayed, such as more detail 
about what the choice means. This can even include effectively displaying the 
instruction manual for that function if desired. 

[00602] The camera system of figure 2-4 as noted can be completely separate 
from the projector, if an auxiliary light source is used. Particularly of interest are long 
lived solid state sources such as white light LEDS or IR lasers. 
[00603] It should also be noted that using the computer controlled projector (or 
other screen) display, a variety of visual cues can be used to signal a function or state 
has been reached. For example, not only can one display a word such as "high" to 
indicate high heat on knob 116, but one can also have it blink 3 times when reached. 
This could correspond to an acoustic or other tactile signal comprising three pulses, or a 
sound generated, as well. The sound too might be coded by three beeps as well. 
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[00604] Also It is possible for the colors or patterns of the words or figures to be 
changed programmably. For example, in the figure 1 case the whole knob, or its 
surroundings could be illuminated through use of an appropriate computer program for 
the display, in bright red when the highest heat position was chosen (with blue, for the 
lowest, for example). And for example, the knob surroundings could be projected on the 
display in polka-dots, if the knob was not in a position that actuated a function (e.g. a 
dead zone). 

[00605] Where desired (e.g. with elderly drivers) the writing on or near the knob, 
might be in large letters, which could even be so large that the words, such as "high", for 
a heater blower speed selected, would need to be abbreviated as HI" for example. And 
in addition, if desired for maximum comprehension, the word HEAT could even be 
displayed in giant letters across the screen, optionally with the temperature setting 
desired, or actual or both. 

[00606] While the projector source disclosed may be used to light one or more 
features of any knob slider, switch or other selection or adjustment means , or a marker 
or other member whose position or movement is proportional in position thereto, a 
separate light source such as a LED or laser light source can alternatively be used. In 
this instance, the camera, can for example see as is known in the art, using an 
interference filter placed in front of the lens or camera array, a filtered image responsive 
only to laser light, the knob mark reflection. This makes the sensing of knob position 
independent of what is being projected, and it can operate with no projection at all. Its 
noted that if the laser (or alternatively for example an LED source) is in the near IR (e.g. 
0.9 microns) the user will not see any indication of this, if desired for further contrast, 
the knob can employ a dichroic mirror reflecting only laser light substantially. 
[00607] More commonly, the projector source itself is used for knob (or other 
feature ) illumination. The illumination while it can be anything desired on the screen, is 
typically chosen to provide a uniform light intensity in the region of the datums on the 
knobs (or other feature) which are being observed in order to determine their rotational 
position. Since the knobs periphery can actually block the light, it can be seen from fig 
3b that it is possible to project light for knob detail illumination purposes which is not 
apparent to the passengers of the vehicle. 
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[00608] The Toffolo invention referenced above has a desirable property of 
making an optimum display. The invention herein goes further and has a programmable 
"fitting" program to maximally fit as large as possible version of the desired video or 
other download information to be displayed on the display, using the fit program to take 
into account the knobs and other tactile control features (not present on the display of 
Toffolo). Or there can be fit programs which keep data away from tactile ridges and 
knobs. And unlike Toffolo, data can be momentarily given precedence over other 
features of normal importance. 

[00609] It is again noted that one can have knobs or other controls whose function 
goes back to standard function after amount of time. Then std function changeable if 
conditions change. For example if rain detected, goes to one where wiper speed and 
selection is one of knobs(where ordinarily wiper issues would have been a secondary 
function). 

[00610] And at night, headlight and viewing aids such as IR may become part of 
standard screen and control surface, which would not be the case in daylight. This is 
commonly sensed by a photo detector today. 

[0061 1] The user can select in some versions if he wants one knob to all the time 
be something. - eg heat speed or heat items constant. For some users constancy is a 
desirable trait. For others it is inhibiting. 

[00612] The driver can signal the system by voice as well as touch. And he could 
even do unusual things such as wiggle in his seat. In this case something could 
change, and logically a seat control could be presented. Or an internal comfort screen 
which could include not only seats but window positions, and other things 
[0061 3] Also note indents (or bumps or other tactile relief items) on the 
screen/control surface of the invention could be selectively lit up at various times, For 
special apps., or only when touched for given length of time or with a push in. 
[00614] The camera utilized can be any commercial camera, such as a CCD type 
capable of producing images which can be interrogated as just discussed. However, 
best results may be obtained in some conditions of operation by using a camera which 
can be addressed on an individual pixel basis, to achieve faster operation. In such 
cameras, typically of the CMOS type, one can interrogate only those areas of screen 
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200 of figure 3a for example, known to contain pixels of interest. For example these 
would in figure 3c be the regions where one might find the location of what ever knobs 
or other control details were on the screen. In this case of knob 201 this would include 
the pixels in an annulus 221 (the same ones preferentially illuminated if desired by the 
projector for example). 

[00615] The regions of interest may include other pixels as well, for example 
those near tactile relief items used to input data. By only looking where data is known 
to be, the camera speed can be increased to even 1000 frames of data pixels a second 
for a screen having only a few control details to be interrogated. This is more than 
ample for any control purposes (typically 30 frames per second is sufficient, the rate of 
change of manually controlled functions being generally slow (though tracking finger tips 
and other functions of the invention to be further described can require higher 
frequencies). 

[0061 6] It should be noted too that once a single knob for example is detected to 
be being changed in its position(by comparison in computer memory for example, to its 
previous position), the camera can often be directed to concentrate image processing 
activity largely in this specific location as the driver in the general case does not usually 
with one hand turn two knobs at once, as an example. 
[00617] It may be desirable in situations of high ambient sunlight in the car 
passenger compartment to use a camera such as the FUGA 15 or the color version 
FUGA 15 RGB ( www.vector-intemational.be ) which is pixel addressable as well. This 
camera has a logarithmic response curve, and can resolve data over huge dynamic light 
intensity range. Thus in a situation where sunlight was entering the passenger 
compartment and hitting the screen 200 in figure 3a right next to knob 201 , the intense 
light would not disturb the camera in its ability to see marker 207 on the rear of the ... 
knob, which knob would act to shadow the sunlight in the annular region in question (or 
over a whole circular area if the knob was opaque rather than trahsmissive in its center ; ,.; ci: 
portion. Other types of cameras, particularly CCD types, can "Bloom" under such 
conditions, and destroy data of neighboring pixels. 
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[0061 8] Similarly ones finger tends to shadow the finger tip seen by the camera 
through the screen from the rear, when touch location or other touch variables are of 
interest. 

[0061 9] One can also control light the light source whether the projector, or a 
separate source. For example, if the camera unit when set at the maximum integration 
time conducive with not causing bad effects such as blooming at near by pixels, has 
insufficient light to determine the presence of marker 207 , the light source intensity if 
desired can be increased for the time needed to make the measurement, thus allowing 
higher signal to noise and/or faster measurements. 

[00620] In addition it is contemplated that the projector could have a function 
(discussed also in fig 13) where it projects infrared radiation just outside the visible 
range such that nothing on the screen can be seen (or a dull red background for some 
people with enhanced red vision), and then the measurements of knob position etc 
taken during the brief period when this situation is the case. 

[00621] A pixel addressing camera such as the Fuga 1 5 can also have application 
to other high dynamic range sensing applications within the vehicle such as the many 
camera based ones mentioned in copending applications, and in my applications 
copending with Peter Smith or Shesh Sharma as co-inventors. 
[00622] Applications can include looking at objects in the car such as children or 
pets, or in a towed vehicle such as a house trailer. Or a camera (or pair for stereo) can 
be used behind the towed vehicle to provide information about vehicles following. OR a 
camera can be located in a pickup bed, to allow the display of the invention to see 
objects in the bed, or in an enclosed truck or pulled boat or anything else where such 
things would aid in vehicle navigation (e.g. backing up) safety or relieving distraction 
influences. 

Figure 14 - 
[00623] Let us now consider another camera'application of importance in saving 
lives. A co-pending application by myself and Shesh Sharma describes a camera 
based system, which may be advantageously connected to the display made possible 
by this invention , which allows observation by the driver, of a baby in a rear facing car 
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seat, usually in the back seat. The camera as shown in that application is located in 
headliner of the vehicle (an application also suggested by Donnelly Corporation of 
Holland, Ml. in their Babyvue product) or in the child seat itself. 
[00624] As shown in Figure 14, data from a camera 1401 observing the baby 1402 
or other children (or pets ) in the car 1400 is processed by computer 1405 , which in the 
simplest case may have no other input indicative of child condition. The presence of the 
child can be determined by seeing the child, determining that an outline or other 
characteristic image of the child exists, and also to compare images and determine 
motion between them, indicative that a child is present (realizing that most children 
make some movements at some time in the seat). 

[00625] Alternatively a simple way to determine that a child 1402 is in the child 
seat 1403 is to look at the child and seat, and if the image of a feature of the seat such 
as 1430 is blocked, the child can be determined to be in the seat (this data can also be 
used for airbag control if desired). In this case it is desirable to see feature 1435 as 
well, not blocked by the child in use, so as to ascertain that the seat is present . Such 
features can be preferably be highly reflective, or otherwise easily distinguishable by 
color or shape . Other means such as weight measurement and change therein on the 
main car seat can also be used for child presence determination. 
[00626] However done, data is used to determine that the child or baby is in the 
vehicle. In addition, one can use data such as that provided by the camera 1401 to 
more particularly determine via computer analysis of image data, that the child is not 
happy (the case if not comfortable) - typically manifested by agitation, which can be 
determined by motion detection between images. This is particularly used to trigger the 
display of this invention to provide a display of the child to the driver. 
[00627] But what if the driver has left the child unattended in .the vehicle, a tragic 
killer of children every year?. The data in this instance , is used by a program in the 
computer to attempt to provide relief and to summon help. Typicajly, such data would - 
be augmented by data from other sensors such as a microphone 1410 to input data to 
a computer based voice recognition program to determine if the baby was crying ( or if a 
pet, a dog barking for example) . Such can be determined by signature analysis, also 
including magnitudes, recognizing the sound is likely to be loud. 
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[00628] Of critical importance is the situation where the child or animal has been 
left in a vehicle during weather conditions over a time period which could cause its 
death or injury. In this case temperature is a desirable variable to provide to the 
computer 1405, for example with temperature sensor 1420 which could be a thermistor 
or other device known in the art, often included in vehicle climate control systems today. 
[00629] Time is also a variable in this circumstance and it may be desirable as well 
to determine a time period from when the car was turned off, or the doors locked (if 
power door locks are present as is increasingly the case). 
[00630] When a condition that indicates that there is cause for alarm, the 
invention herein first determines same, namely that the car has been left unattended, 
that the temperature is in a dangerous zone for a period of time, and that a child is 
present (and optionally unhappy, though this does not have to be an essential criteria). 
If these criteria are met, the invention first puts the windows down slightly (Assuming 
power windows, and a high temperature condition). If a cold temperature condition is 
determined, the windows are left up (or put up, if down) and the horn and lights can be 
sounded, just like a burglar alarm. And if equipped, a signal to the driver can be sent 
via wireless means. 

[00631] The hot weather situation is the most dangerous typically. In this case it is 
contemplated that if children are determined to be in the car at all, that the fan of the 
vehicle would be engaged at a good speed setting. Then after some short period, the 
windows would be lowered a little ( or a sun roof opened) . This period of time could be 
zero if desired, and a warning made to the parent even before leaving the vehicle- like 
leaving ones key in the lock today. 

[00632] If the condition persisted some programmed time, also in consideration of 
the temperature conditions and their effects, and in consideration of the child's state 
(e.g. present, agitated, crying), then emergency measures such as horn and lights 
could be engaged . 

[00633] The camera data as noted can come from cameras in the car or; in the 
baby seat. 

[00634] It should be noted that the ability of the invention as disclosed particularly 
in figure 10 to provide a means by which the driver can designate regions of interest 
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greatly eases the stereo pair image matching problem, where features in one image are 
determined to be the same feature as identified in the other image of a stereo pair, in 
order to determine range by the offset in the image location. 

[00635] The use of stereo cameras in vehicles is for example described in recently 
issued US Patent #6396397 by Bos, et al. assigned to Donnelly corporation. Such a 
device is limited in scope by the need to automatically match - difficult to do reliably if 
done in a totally passive manner without active projection light source aids. While car 
taillights or headlights having known spacings, colors, or other characteristics can aid in 
the matching, this is generally effective only at night and only when no confusion exists, 
as on a country road. 

[00636] The invention allows the field of view of the cameras to be vastly limited to 
the region around the object designated, and further allows the image edge and other 
features of that object to be locked onto as the features to be matched. Since the 
camera views are relatively parallel l(i.e. the baseline between the cameras is relatively 
small relative to the distance to the object) the views have similar image characteristics 
( under normal daylight lighting conditions) though of course the images of the object 
obtained by the two camera are offset due to the baseline . 
[00637] It is further noted that computer 1 20 or 1405, or other computers of the 
invention used to determine data from cameras used to determine information from the 
screen, from inside the vehicle (e.g. fig 14) , or outside the vehicle (e.g. fig 10b) , can 
have incorporated a memory such as solid-state RAM Memory or disc, which can 
record images obtained. These images could be recorded with a rolling 30 sec 
window, resulting in 900images if obtained at 30hz , and could allow one to diagnose 
after a crash what happened, at least from the video data. Brake and other data might 
be stored as well. The computer and memory could be protected like a flight data 
recorder for this purpose. Clearly more images might be stored if -desired, for example if 
higher data rate cameras were used. ■ 
[00638] It is noted that this particular application is aimed primarily to provide a 
much needed control interface and information system for the driver of a vehicle. 
However others can make use of the beneficial aspects of the invention as well. 
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[00639] For example, the invention can be used for other automation or control 
purposes, such as machine tool control . For example a readable card of the invention 
such as disclosed in figure 1 1 can be shipped along with some semi-finished parts to a 
CNC machine tool station (or other working station) which is to finish the parts. The 
card is put into the slot of the control and both the program needed for that part, and 
the specialized operator control touch elements needed for that part are accessed by 
the machine computer and used to perform the operation. 



Figure 14 

[00640] As shown in the diagrammatic side view of figure 1 5a is another example 
of the invention. A center stack of an instrument panel (not shown) is equipped with a 
large (for a vehicle e.g. 10x12 inches) rear projection screen panel 1505 preferably 
made of non shattering light transmissive plastic, having in this example a myriad of 
small glass beads 1504 on its back surface which serve to widely diffuse the light 
incident on them, in a manner creating minimum backscatter . 
[00641] The beads are encased in black plastic, with only the tiny spherical 
surface near their focus (which due to their index of refraction of approx n= 1 .9 is 
approximately at their surface) protruding from the black matrix holding them . This 
design increases contrast of the display, and is also useful for keeping light from the 
drivers side from entering the optical system of the invention, and for shielding the inner 
workings of the device from view of persons within the vehicle. A typical commercial 
screen of this type is 3M "Vikuiti" XRV type NP, which is 1/8 inch thick approximately, 
with glass beads 0.002 - .003" in diameter. 

[00642] Thicker or thinner screens can also be produced, which can be made by 
laminating the base film containing the beads and blackening material to another 
material, either flat or curved, and typically of lexan or polycarbonate plastic. For the 
bead type material to work as designed, the open aperture of the beads should be : 
toward the projector . 

[00643] A version of this having a fresnel lens incorporated is called "Black bead" 
and is made by Dai Nippon printing co in Denmark. 
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[00644] Alternatively screens having less dispersion vertically than horizontally can 
be used, as well as ones tailored using holographic diffusers or gratings or other optical 
elements such as lenticular arrays to disperse light preferentially to the range of viewing 
directions of driver locations 

[00645] The display screen/control surface extends out of the plane of the 
drawing. While shown flat in this drawing, the screen/control surface in many motor 
vehicle cases is curved for stylistic or other reasons. This is particularly true since the 
display screen/control surface is desirably large and needs to fit in stylistically with 
other parts of the instrument panel which are generally curved themselves. The 
curvature can be either convex, or concave, or compound, and can be of irregular 
shape, and with portions even staggered in the z axis . (see figure 18) 
[00646] A computer controlled display projector 1 51 0 controlled by computer 
1 520 illuminates the screen. This projector may be on axis as shown or located at 
angle to the screen normal in either plane. 

[00647] Video imagery or Computer data including messages and other 
communications 1521 down loaded to the computer 1520 from external sources by 
known means, or from sources 1522 within the vehicle itself, can be thus projected on 
the screen. 

[00648] In this embodiment, knob 1 51 5 , and if desired, further knobs or other 
control details not shown for clarity, are mounted, for example with pin 1506 directly to 
the screen 1505 so as to be rotatable thereon to various positions which are sensed (for 
example by electro-optical means such as camera 1517 which looks at points such as 
mark 1518 on the back of the knob 1515 or other wise related to knob rotational 
position) and reported to the computer 1520 which in turn calculates the knob position, 
functions or other data and executes the control function desired by interfacing with the 
cars electrical and control system and further causes the display device 1510 typically 
illuminated by a white light xenon or other lamp to project suitable information 
concerning same onto the screen. This information is typically data concerning the 
knob position and the setting resulting there from. The same camera can view and 
provide data used to determine the state of a host of different tactile physical selection 
or adjustment means such as knobs sliders, dials, or switches on the screen. In 
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addition, their various positions and changes therein can be analyzed nearly 
simultaneously by the computer 1520 and appropriate control and display responses 
made. 

[00649] Alternatively the camera can observe a datum 1 521 on the back side of 
the screen and rotating with the pin as the knob is rotated. This gives the best contrast 
(as the beads or other screen diffusers don't interfere) and in this case a retro-reflecting 
datum such as 3M Scotchlight 7615 or a corner cube can be used if desired (which 
otherwise has difficulty functioning with the beads) , The image field of the camera 1 502 
for this knob situation is shown for the knob in the initial position 1512, and for rotation 
90 degrees counter clockwise from this position(as viewed by the camera) 1 513. 
[00650] The knob does not have to be held by a pin, and can be fastened to the 
screen by any appropriate means, including adhesive . And it may be held from the 
outside circumference by a bearing , leaving the whole inside surface of the knob face 
clear for data display if desired. 

[00651] In a similar manner other physical details can be provided on the screen 
such as sliders, switches and the like. 

[00652] One useful projector display device 1 51 0 are those having MEM's based 
Digital light processing (DLP) chips by Texas instruments. Another is a system such as 
employed in the 3M brand 7640i projector, having three LCDs each modulating a given 
color, R G or B. It should be noted that for the automotive application, high resolution is 
not required and relatively inexpensive MEMS or LCD "light valve" chips can be used (in 
comparison to HDTV applications, for example). Even less expensive are the LCOS 
(liquid crystal on silicon) chips on the horizon, which, like the DLP chips, operate in 
reflection 

[00653] Today projection devices such as these used for presentations in; 
conference rooms are illuminated by white light projector lamps which are very hot , 
high voltage, and expensive (e.g. $400). It would be desirable to have another source^ 
for automotive use. But such conference projectors are meant for big areas (e.g. for' 
presentations) and the vehicle only requires a fraction of the area to be illuminated (e.g. 
10x 10 or 100 inches squared, vs. 40x50, or 2000 inches squared - or more- for a 
projector . If the ratio is 20 times, the light power can also be so reduced. This then 
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allows one to use smaller bulbs, run bulbs In de-rated manners, or the like in order to 
achieve minimum cost and /or maximum bulb life- an important issue for automotive 
use. 

[00654] More appealing would be solid state sources such as diode lasers or 
LED's as an illumination source. We have run a test with a LumaLED brand 
superbright red orange LED , which produced somewhat acceptable results over a 
smaller area . It was 55 lumens vs. 800 to 1000 of sample small projector such as a 3M 
model 7640. White as well as monochromatic colored LEDS of 1 20 lumens are now 
available as well. 

[00655] One generally needs to even out the light field of the semiconductor 
sources to get best results when using the above projected image modulating 
technologies. This can be aided by putting a suitable holographic diffuser in the path of 
light from the LED. These can be ordered from Edmund optical co, with various 
dispersion angles as desired. 

[00656] Fig 1 5b illustrates a combined LED source 1525 which can be used in 
place of conventional source 151 1 . It has three colored LEDS, Red Green and Blue 
(labelled R, G, and B, respectively) whose outputs in this example are combined using a 
high density fiber optic bundle 1 530, such that the emanating light field 1 531 is 
substantially the same regardless of which LED is being illuminated. With some 
alternative time sequential combination approaches, the blue field is projected, then a 
green field, and finally the red (or in some other order), with each modulated over the 
field by the LCD, LCOS or DLP device as the case maybe. The eye combines the three 
fields if all projection is completed within known eye integration intervals on the order of 
0.05 sec. 

[00657] Note that another problem with solid state sources such as diode lasers or 
LEDs using combinations of colored beams of same to generate a desired color mix, is 
the variation and/or degradation thereof with respect to time and temperature. In this 
case one may wish to use a solid state color camera 1517 (rather than the monochrome 
one otherwise possible to use) to in addition monitor colors of light projected periodically 
in test patterns and to adjust accordingly the relative projected intensities of the 3 
colored LEDs (RGB) . 
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[00658] A fourth LED such as 1 526 operating in the infrared can also be used to 
illuminate the screen and control surface to provide sensing of physical details and 
fingers irrespective of the projection illumination visible to the eye. In this case sensing 
can be done with a camera having for example having a filter having a pass band of 30- 
50nm wide at for example a typical IR LED wavelength of 905nm (thus passing light 
even with small shifts of wavelength due to LED temperature variation and other 
factors) . 

[00659] Or as disclosed in a copending application a DLP device itself to perform 
the scan of the signal to a detector substantially coaxial with the DLP device. 
[00660] Even simpler systems can have only two colored LEDs for simplicity, at a 
color palette sacrifice in ability to generate a wide range of colors. However for the 
instrument panel use a complete color palette is not necessary, though desirable for 
maximum customer value. 

[00661] Now illustrated are further physical details and touch features of the RTD 
disclosed in previous applications 

[00662] The knob datums used in one illustrative experimental example, shown in 
figure 15c, were simply aluminium portions of the knob housing 1530 itself, which had 
been blackened (and thus made non-reflective) in all regions but where target datums 
were wanted. The knob as shown is held from its outside in bearing race 1531 attached 
to the screen 1529. The knob is hollow which allows data to be projected to the screen 
in the middle of the knob by projector 1 532. The rear of the knob facing the camera 
1 538 is shown having in this example three targets 1 533-1 535 forming an isosceles 
triangle whose base line was between the two closely spaced targets, and the axis of 
the knob being defined as a line perpendicular to the base to the third target 1 535 at the 
apex. This arrangement allowed the pointing angle of the axis to be determined easily 
by processing the image obtained by the Sony C0TV camera used. This image was 
first acquired, by a Matrox "Genesis" image processing board, apd then using the 
Matrox Mil 4.0 image library, processed first to find the blobs represented by the targets, 
and then the triangle axis calculation was performed. In another and preferable version, 
an added step including first subtracting the image obtained from a previous image 
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taken under another condition not representing the instant situation, and then finding the 
blobs and the triangle calculation. 

[00663] Once the knob location was found, a lookup table relating to the angle 
obtained was entered to find a value of the variable desired to be projected. In one 
case the projector was controlled to project a line on a radio dial, indicative of where the 
knob, in this case representing the tuning knob, was in terms of frequency. In another 
case, the same knob was used to control the projector to project a heat bar of different 
color and length dependent on knob rotational position. In each case output signals 
were also provided to control the physical radio or heat mixing door of the vehicle HVAC 
system. 

[00664] Figure 15d illustrates another form of sensing, this time of the pressing in 
of a button 1551 into screen 1552. etc as shown when the button is pressed in, 
reflecting member 1555 is pushed to a new position (dotted lines) and in so doing 
directs more light from projector 1 560 to camera 1 561 . The image when thresholded 
to an intensity between the two values, allows a determination of ON (above threshold) 
or off (below threshold). 

[00665] Figure 15e illustrates an alternate method of determining button push 
which can optionally also be proportional to actual z position into the plane of the screen 
(in other words having a range of positions, not just on-off). For illustration purposes a 
row of switches 1 565 is arrayed along a horizontal line into the plane of the paper. And 
a second row 1566 is also provided located below the first. Also illustrated is that a 
portion of the screen 1567 containing the second row can be spaced as well in the z 
direction from the other portions, as long as sufficient depth of focus exists for either 
projection or detection functions. Camera 1570 controlled by and interfaced to 
computer 1571 is used to view the switches. A projector or other illumination means 
1 572 is used to provide data on or adjacent each switch as to its function, with said 
projector controlled as well by computer 1571. j 
[00666] The angle Theta between switches in row 1 565, and the camera axis 
allows one to measure of position of the switch in the z direction into the screen by 
triangulation. For example when switch 1565 is in its outward position, the image 1574 
of reflecting datum 1575 on camera array 1576 is as shown . When pushed in a 



119 



distance "D" to the location shown in dotted lines, the image position is at 1577, due to 
the included angle Theta. The distance d, in the image, gives a measure of D by 
known triangulation principles. 

[00667] One an use a second camera as well such as 1578 shown in dotted 
lines. This allows one to see data from knobs and other details from more than one 
vantage point, and in some cases with higher resolution when both cameras data is 
combined. 

[00668] It should be noted that the position of a knob pin shown in fig 1 5a can also 
be monitored in z if it is pushed in, much like the buttons of fig 15e, for example in order 
for the person turning the knob to register that he has reached a desired location, or to 
query the computer for what the present location is for example 
[00669] A knob having three targets like figure 1 5c can also register an inward 
push. For example consider fig 15f in which such a 3 target set 1580 is on the side of 
the screen facing the projector, and rotated via pin 1581 by turning knob 1582. 
[00670] When the knob set is pushed in, the target 1 580 moves to position dotted 
lines. While the image on the array is more complex than that of fig 1 5c, it can be 
solved to decouple rotational position from z axis position in the direction toward the 
projector. As before the camera is located at an angle theta to the screen normal to 
allow resolution of z by triangulation 

[00671] Taking this one step further, one can construct a multi-axis knob such as 
1 585 shown in figure 1 5 g. Because the position of the target image in the xy field of 
camera (not shown for clarity) is changed as you push side to side or up and down , but 
goes to points not allowed for normal knob rotational movement, one can solve for the 
change in position of the target such as 1586 due to rotation as opposed to 
displacement in the x or y direction of the screen. In the case shown, X and Y motion is 
created as shown, using a gimbal 1587 which converts an angular change caused by 
pushing the knob right or left (or up or down) from its center in to a movement in the 
field of the camera (1594 shown in dotted lines) . For example as the knob is pushed 
down in the y direction to the new deflected position 1590 shown in dotted lines, the 
target 1 586 moves up in the y direction from position p to position p*. 
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[00672] Note that where desired in the embodiments herein, pixel addressable 
cameras can be used to speed up datum tracking using such techniques such as 
described in co-pending applications incorporated by reference herein. With such 
cameras one only need scan the regions of the image where information is expected. 
Generally speaking, for example if one turns a knob, and the knob image is being 
observed to change, only the pixels that relate to this knob need be scanned in 
subsequent camera views unit the knob stops turning . This is because the driver 
cannot in general turn two knobs at once, or use a touch screen while turning a knob. 
This leaves a lot of computer power available to process other data of interest such as 
that coming from external sources such as TV cameras or Lidar sensors for example. 
[00673] The camera shown can view the datum's and fingers from whatever angle 
is satisfactory. Using a beaded screen like 3Ms there is an approx +/- 30 degrees 
angular attack with respect to the normal that one can get data from datum's on the 
drivers side of the screen thru the beads of the screen. This is not a limit if the datum's 
are on the camera side of the beads. 

Figure 16 

[00674] Figure 1 6a illustrates a front view of apparatus along the lines of fig 1 , in 
the form of an embodiment of the invention employing knobs of fig 1 based on a 
"traditional" Radio layout 1600, located on screen 1601 in the center stack region of 
the instrument panel. 

[00675] In fig 16a example, the physical control details on the screen itself are the 
two knobs 1610 and 1611, and four physical pushbuttons 1620-1623, such as shown in 
fig 15 . (Note other numbers and sizes of knobs and pushbuttons may also be 
employed, and the buttons can be virtually projected and actuated by touch if desired, 
rather than physical pressing in). The projected image portions on the screen are the 
surrounding lettering and graphics, and the radio dial numbers 1 625 and indicator -■ • 

needle 1 626. This is like many 1 950's car radios, and one can even project graphical 
representations on the screen which can even be copied from actual radios of the era. 
[00676] Projected graphics and alphanumeric characters are also used to 
illuminate the knobs, such as Vol and Tune shown. This illumination can be next to the 
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knob, or if the knob is constructed as shown with a transparent center section , right in 
the middle of the knob. 

[00677] Similarly the buttons can be so illuminated with projected radio station call 
signs, such as WXYZ, or the frequency, or some other delineator. The knob centers 
can also contain projected information if desired, as can all regions around the various 
controls. As shown the function of the knobs, for volume (VOL) and tuner (TUN) are 
displayed. 

[00678] A typical height of such a device might be H = 12 inches, and width W = 
1 0 inches. The region at the top of the screen most in the driver's field of view is 
desirably reserved for key images such as those obtained with backing up cameras or 
other important data. This area can also be where virtual controls operated by touch 
are used as desired. 

[00679] Figure 16b illustrates the case where the function is changed from a Radio 
to a heater, more correctly today called Climate Control or HVAC in the trade (heating 
ventilating and air conditioning) . The buttons and knobs are knob re-labeled in the 
projected data to suit the new function. For example the left knob now is the 
temperature (TEMP), and the right knob is now fan speed (FAN) . In this example, 
pressing a button in (either a physical real button, or a virtual projected button as 
desired) may be used to control air direction choice as one example. 
[00680] Entertainment and Climate are the main control sections found in the vast 
majority of vehicle centerstacks today. Since this one reconfigurable control of the 
RTD invention accommodates both, this leaves more space for an associated Video 
display, and/or for larger knobs and lettering, and other functions too, as desired. And 
they are provided in a manner very similar to today, that is with physical knobs and 
buttons, in the usual expected places (e.g. with knobs on both sides of scale or other 
display). The only difference is that the entertainment function is reconfigurable with the 
climate. 

[00681 ] But it needn't stop there. For example fig 1 6c illustrates the screen 
centerstack with neither radio or heater projected- in other wards in its empty state, with 
the system off (except for a small ring projection at switch indent 1651 ). Clearly anything 
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else desired may also be projected which could be controlled with the two knobs and 4 
physical buttons shown. 

[00682] Where buttons are virtual projected types, small relief details permanently 
on the screen such as ridges or grooves at the bottom of the button region as taught in 
copending applications are useful for finding the button location by feel. The buttons 
can be chosen to be projected at the desired locations where the relief details are., for 
whatever function radio, heat etc is desired. 

[00683] To switch from one function to another (e.g. Radio to Heat), one can use a 
variety of means. For example, on the steering wheel one may include switches for the 
various function groups- climate, entertainment, comfort, etc. Pushing or scroll dialing or 
otherwise actuating a switch would switch the function to a heater for example from 
what ever it was (e.g.., a radio). Or one can use a simple touch switch on the steering 
wheel or elsewhere to scroll through the functions desired. In addition or alternatively 
one can use voice recognition to do this, by just saying "heater". 
[00684] In addition or alternatively, function switches can be on the RTD itself, and 
can be hard physical details like the buttons, or in the case shown, "soft" using 
projected icons such as 1650-1653 on the RTD screen 1660 which is equipped as 
disclosed with touch sensing capability to respond to finger touches thereon. These can 
be used with relief features to aid their finding by touch as has been pointed out in 
referenced applications . As shown, round switch icon 1651 is shown illuminated after a 
touch by a driver to select it (the function selected such as climate, can also be 
displayed). Conversely the illumination of icons can be under computer control and 
selected to only illuminate those for which action may be needed. 
[00685] Other external switches can be used too, for example on the left or right 
arm rest. This is particularly easy if a limited number of screens are used, for example 
3 or 4 (climate, Entertainment, comfort, and safety, as will be described below). 3 u . ' 

[00686] The use of soft functions allows the maximum screen space to be freed 
up, for example to facilitate display of TV images and other information. One such 
image is the region behind the vehicle, taken with one or more TV cameras as 
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previously noted. This is ideally displayed in the region at the top of the screen along 
the drivers line of sight. 

[00687] Selection can also be done with a multi-axis knob, which can be pushed in 
to select in sequence, or a 4 axis knob as disclosed above to allow one to select one of 
4 displays simply by momentarily jogging the knob to one of the 4 quadrants. Thus the 
radio volume knob for example in the figures above could also be a selection knob for 4 
different functions- e.g. Radio, Climate, CD/DVD and telecommunications. 
[00688] A typical scenario for operation is for the driver to unlock the car with a 
radio transmitter which then signals the computer of the RTD to display in region 1 670 
shown in figure 16b the view behind the vehicle taken with camera or cameras not 
shown looking rearward. This view is displayed until forward gear engaged or some 
manual override occurs, to give the maximum time for the driver to see the image of 
things or persons behind him . 

[00689] Also in the initial condition, the Climate control display such as that of 1 6b 
also is activated. Settings are made, or not made, by the user, and then after a few 
seconds of inactivity (variable by user setting) the system may default to the 
entertainment screen of figure 16a. unless the climate controls are being worked by the 
operator. 

[00690] Other manual overrides would be to cause an additional screen(s) to be 
projected and set by the operator. 

[00691] For those who are audiophiles, the whole region 1 670 above the 
radio/heater section (in many cases not having physical controls but rather being 
reserved for video and data displays and virtual controls) could for example, be 
devoted to auxiliary audio controls such as for equalizers, and mixers. This could be 
presented when in radio mode, or when in other modes such as shown where heater 
(climate) function is shown in fig 16b using the physical controls while radio controls are 
in the upper virtual portion, in this case audiophile controls for example using virtual 
sliders 1680-83 to mix sounds from several sources . These are ideally provided in a 
virtual manner using the touch sensing capabilities of the invention, but can alternatively 
be selected with a multi-axis knob such as shown above or another type of selection 
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device such as mouse, joystick or the like. Or alternatively provided as hard physical 
details. 

[00692] Another centerstack control combination and screen/control surface 
design is shown in figure 16d where one could have the instruments which are shown 
displayed in the instrument cluster in front of the driver , alternatively displayed the 
virtual display area of the RTD in the Center stack. 

[00693] Another point illustrated in figure 1 6 b is that one can have both heater 
and radio on the screen at once. In this case with the top one being virtual and the 
bottom one at least partly physical (permanently in place) controls. Or one can 
duplicate radio for example in both regions. As might be desired if you want to drive 
safely without looking down at all, and put the key radio controls (for example volume 
and tune) in a virtual sense right at the top of the screen, for example while driving in 
dense traffic. You could make them large size too. 

[00694] It should also be noted that figure 1 6a-c illustrates what can for example 
be a very nice useful control system for use in the home, to control appliances and 
entertainment equipment, typically from a location on a kitchen wall where it can be 
easily accessed by a housewife or other person. It is once again, intuitive and easy to 
see and use. Here again the familiar metaphor of traditional car radios and heaters can 
be used, if desired, ( to operate the entertainment , heating and appliances of the home) 
with easy to operate large controls preferred. 

[00695] In the home, the screen and control surface can be much larger, if 
desired, than would be possible on a vehicle instrument panel. For example the 
buttons (physical or virtual) 1620 -1623 could be used to switch from one appliance to 
the next, with the two knobs then reconfigured in the computer and the display to be 
the key controls for that appliance or entertainment device . 
[00696] In the home application, the upper display area 1 670, when not used for 
virtual controls, can show TV programs from a suitable source, such as a cable 
connection. With suitable optics, not shown and a. mirror, the projected information can 
even be routed outside the enclosure (through a hole in the side, say) to project for 
example on a nearby family room screen . Or the whole RTD unit might be demountable 
from the wall (or other suitable location), such that it could be put on a table and used 
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as a projector. In this case, one might just remove the front screen/control surface (held 
on in an easily demountable manner as with wing nuts for example), and just adjust the 
focus of the projection lens to cause the projected information to be thrown on a far wall. 
[00697] It is noted that this function, of routing projection data to another location 
can also be done in a car, for example when stopped, so as to project DVDs in a 
manner similar to the Drive In movie of old. In a minivan, the screen could pull down 
from the roof, such that passengers in the 2ond and 3rd seats could view the movie 
projected from the RTD in the instrument panel. 

[00698] Information for controlling an appliance can be communicated over a 
home network, which may in the future be wireless. This is perhaps the most economic 
solution if one were to install the RTD control in an already built home . In that case 
each controlled appliance or device would have the receiving and transmission 
capability to communicate with the RTD. This could be bluetooth or other known 
wireless conventions. 

[00699] Alternatively, completely different control panel layouts to that shown in fig 
16 can be employed. 

[00700] The fact that I have just used figure 1 6 to illustrate both an automobile 
instrument panel and a home automation control, is a major advantage of the invention. 
If such commonality can be achieved, not only does cost drop, but usage and learning 
becomes easier. The famous VCR programming difficulty in the home could be 
mitigated if the control layout mimic-ed something one was intuitively familiar with. 
[00701] In looking at the curved display screen/control surface shapes possible 
with the invention, for example as illustrated in figure 18, it is also possible to envision 
the invention built not only into the flat walls of homes or offices, for example, but into 
various elements of furniture as well where it could also provide a stylistic departure 
from the usual control panel flatness. Even a small curve can make it distinctive. And 
as noted, the curve can be compound in two axes fbr added distinction. ,, It can even.be 
re-entrant, for example S-shaped. 

[00702] Since the screen can be rugged, one could even sit or lean on it, 
assuming the rest of the housing could support the weight. And because the system 
has a light source in it, this light source can be used when desired to simply act as a 
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room light. In this regard the projector can be programmed with pleasing colors, which 
can also change in power and color to suit the mood desired. 
[00703] Other aspects of the invention also lend themselves to innovative 
furniture. Where rear projection is used, features of the application can even protrude 
through the screen of the invention, such as for example a pole holding an awning, if the 
screen and control surface formed the base, since the pole could miss the key 
elements, and the light projected around the pole. Since the screen can be irregular, 
the pole (or another feature such as vase ) could be indented into one side of the 
screen for example, like the vent 2701 of figure 27. 

[00704] While it may be fanciful to think of sitting on ones control system when it 
isn't in use, such stylistically attractive controls can be of considerable interest in 
designing appliances for the home for example. While the RTD is well suited to 
controlling a group of appliances, for example with a wall mounted unit as discussed, it 
can also be designed into the individual appliance itself. This is made possible in the 
future, by vastly less costly component costs. And this appliance can thus have an 
irregular or curvilinear control panel as taught herein. 

[00705] The TV set of the home, can be an free standing RTD, as opposed to a 
wall mounted device as described in figure 16. Here too, the ability to have not only 
controls for other functions incorporated with the TV, but also to have it be a stylistic 
piece is of major interest. However, if the curved screen is to be used this assumes 
that people would want to watch a TV Screen that was not flat . This may not be the 
case, and if not, the stylistic issue may be to provide a screen surface flat over a large 
area, with controls and touch functions or other things off to the side in regions which 
can if desired by curved or irregular. 

[00706] If one considers a home RTD TV with controls on the side, the control 
portion could be not illuminated by the projector, during normal operation of the TV 
display portion. Thus the presence of the controls would not be distracting to the 
viewer. Then when control is desired, they could be illuminated as taught herein. It 
could be in this case too, that only the portion off to the side would have a touch sensing 
capability, with the main TV display portion not requiring same. However, it could also 
be so equipped. 
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Figure 17 

[00707] Figure 17 is a block diagram illustrating control of various devices by a 
PC based version of the invention, in which the projector unit 1770 is controlled by PC 
1771 in response to programs resident thereon. Image data 1774 from Camera 1775 is 
processed 1778 by Matrox MIL software resident in the PC. Data from the processing 
is analyzed 1780 to determine position of fingers or physical details as for example 
described above, and from this a determination 1784 is made as to what control actions 
to take, which action data 1785 are used to control devices such as motors, relays or 
other devices , and to generate appropriate signals to change the display accordingly. 

Figure 18 

[00708] Figure 18 illustrates an irregular shaped, curved display panel and control 
surface 1800, in a configuration suitable for some instrument panel center stack 
applications. The display surface is shown in this example as curved, irregular in 
shape and may if desired have a ridge, disc, or other shaped protrusion or depression 
such as ridge 1805 extending outward or inward of the surface, on which information 
may also be projected or sensed (as long as sufficient focus can be maintained, usually 
not difficult with relatively small depths/heights ). The screen is illuminated by a 
projector 1810. Physical details and touch points on the screen are sensed as 
described herein, for example with camera 1820. It should be noted . The irregular 
shape shows a "dog leg" 1818, which allows the screen to be cut out for the drivers right 
leg clearance if it is desired to utilize all available instrument panel real estate where it 
can be seen and controlled by the driver. 

[00709] While beam paths may be folded via mirror elements to allow less depth to 
be used in the instrument panel, with the convex curved screen shown it is easier to 
provide optical elements such as 1811 and 1821 capable of wide^ngleHHuminatiomand 
imaging of the screen (while maintaining acceptable degrees of focus). In many cases ,. ■■ 
therefore the need for mirror elements is obviated, resulting in less cost, and a saving of 
some other space aspects. And too with fewer optical elements it is easier to utilize the 
total RTD device in its housing 1850 as a plenum for air distribution from blower 1854 
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through vents 1855 that might be provided in the screen 1800, without suffering 
condensation problems caused by cooled air passage. If this is a problem slightly 
heated windows for example can be used to shield the projector and camera optics. It 
is noted that vent 1855 can be clear plastic such that data can be projected on it (if 
diffusive), or seen through it to a diffusing screen such as 3M Vikuiti behind. 
[0071 0] In the auto application (and others) it is desirable to have all the 
components of the RTD Packaged into a housing such as 1850 which can be easily 
inserted into the vehicle in question. In this case the minimum possible connections 
would be desirable from the RTD to the vehicle for power and input output cabling. It is 
envisioned that the computer, projection, sensing (camera or otherwise) would all be 
contained within the housing in most cases. However, the screen/control surface might 
be fastened on later, if all that was necessary was to program the system to deal with it 
(in other wards the projector and sensor would be focused and set up for dealing with a 
range of different screens and control surfaces that might be installed. This could even 
occur a dealership. 

[0071 1 ] Mitsubishi has developed a rear projection device having a flat screen 
using a Texas instruments DLP light valve which appears to use at least an 80 degree 
field of view as taught in their recent patent USP 6577455. This unit is only 10 inches 
deep with a 60 inch diagonal 4:3 aspect ratio screen, and is ideal for mounting in or on 
the wall of a house or building. 

[00712] With convex curved screens image formation is easier at least in the plane 
of most curvature, realizing that the instrument panel is typically can be much more 
curved in the vertical plane than in the horizontal, as illustrated in fig 1 8. Thus very wide 
angle projection systems can be used in the vehicle - especially since the application is 
less demanding regarding resolution and color fidelity than HDTV for example. If wide 
angle optics are used with a mirror, the depth in the fore-aft direction of the vehicle can 
be just a few inches. 

- [00713] Figure 1 8b illustrates a concave cross section which can also be used 
rather than the generally convex shape of figure 1 8a. Even combinations of sectional 
shapes can be used, as long as the projector image can be sufficiently in focus for the 
intended purposes. The camera image focus is typically not as important as the datums 
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on the knobs or finger images and the like can generally be quite defocused and still 
register. In fact some degree of defocusing can be useful in some cases to average 
over a number of pixels allowing more subpixel resolution in the vision processor. 
[00714] Also illustrated in figure 1 8a is a unique capability, namely that in region 
1830 (dotted lines) the camera 1820, or even a second camera not shown, can be used 
to see two measures of person identification. In the first the finger print is read of the 
person pressing in on the screen in this region (which region can be projected as an 
appropriate region for such if desired under control of the projector, and can be varied 
from time to time to avoid buildup of finger data on the screen which could be lifted off 
or otherwise used by a potential thief), and in the second, a finger gesture on the screen 
of the person in this region can be determined by analysis of a sequence of finger 
locations, or movements, or the juxtaposition of two or more fingers. Such a gesture is 
easier to detect (due to limited resolution when only one camera is used for the whole 
screen), can be taught to the camera-computer system, and can be very simple or as 
complex as the owner of the vehicle might wish. 

[0071 5] There are many possible sequences of finger gestures that can be used 
as well. If a time delay was built in between each try, this could be an effective theft 
prevention device. And some gestures would require a degree of dexterity which could 
guard against drunk drivers as well. 

[00716] Gestures could be rotational or pinch gestures with thumb and forefinger, 
sweep gestures with ones forefinger in any prestored direction. And so forth. One could 
also draw with ones finger ones name or a portion thereof. Any or all would serve to 
identify the person, at no added cost. 

[00717] For example, illustrating a gesture 1848 of this type could be wherein the 
user draws a "Z - like" gesture on the screen, shown in dotted lines. This then can be 
followed with another gesture if desired for example another letter or some other sign. 
The gesture can be remembered after being taught, or a sophisticated hand writing 
recognition program can be employed such as in a tablet PC to detect actual writing 
with ones finger. This can also be used to communicate or do more in commanding the 
control system as well, as an alternative to voice input for example. 
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Figure 19 

[00718] It is also possible to use electro-optical sensing to determine movement of 
the screen related to force caused by touching, and in so doing determine either that a 
touch has occurred, or if more sophisticated processing is used, to determine where it 
occurred in the area of the screen. 

[00719] Figure 19a illustrates optical sensing of touch occurrence or position from 
quasi rigid body movement of the screen/control surface. As shown screen member 
1901 (here illustrated as a flat screen for convenience, though any shape is possible) is 
mounted to the instrument panel housing 1902 at its four corners, 1905-1908, using 
compressible supporting members as further described below, which allows 
displacement under the load caused by the touch of for example, finger 1910 directly on 
the screen 1 901 , or alternatively on a physical detail such as knob 1911. The screen is 
observed through a suitably sized opening in the instrument panel housing by TV 
camera 1920 whose image information is processed by computer 1921 . Both the 
camera and computer can be the same one as used in other embodiments herein for 
knob position and the like. Data is projected by projector 1915. 
[00720] In one version of the invention of this type, the location of the screen 
edges can be sensed using the method disclosed in fig 15 above . And fixed datum's 
on the screen such as a reflective spot 1922 on the rear surface of the screen can 
additionally or alternatively be sensed to move in their movement in the z direction 
under touch load, and used to determine said condition. Where a single camera sees 
movement at all points required, and is the same camera as used for knobs and other 
physical details, this typically results in a very low cost system. 
[00721] If the supporting members supporting the screen are designed to 
constrain movement to be effectively only in the z direction, it is relatively simple to 
solve for the position of touch, assuming all members have the same resistance to 
force. 

[00722] When used in automotive applications, It is desirable to have the 
supporting members be as stiff as possible, to avoid excitation of the screen due to 
vibration of the vehicle . This in turn implies small displacements under the load of 
touch, following Hookes law (assuming the support member is completely elastic). And 
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thus the resolution of displacement by the camera system has to be quite good. Or 
multiple cameras used each covering smaller zones, for example encompassing just 
one of the supporting members. Alternatively other types of sensors of displacement 
can be used, but this adds cost. 

[00723] In one example, shown in figure 1 9b, a single camera 1 920 is used to 
observe all four corners of the screen 1 901 , of which only one 1 940 is illustrated in this 
view, which is supported by support member 1941 in this case comprised of a coil 
spring 1945 acting in a housing 1946, secured to instrument panel 1902 . When finger 
1910 presses on the screen the z axis component normal to the screen and parallel to 
the housing acts to move the datum's observed by the camera 
[00724] Two types of datum's moving under load are shown in this drawing. First 
is the reflective or otherwise observable datum 1960 attached to the end of rod 1 961 
secured to the screen and passing thru the center of the coil spring. As noted relative to 
figure 1 above, a z axis movement of 0.010 inches can be detected using a 1000x1000 
pixel element camera to observe the complete screen of 10x10 inches in extent. In 
order to see accurately, the datum should be large enough so that many pixels of the 
camera can be used to determine its location to sub-pixel resolution, in which case a 
movement of as low as 0.001 inch can be determined. See for example, US Patents by 
the inventor and his colleagues for details on methods to achieve this. 
[00725] The datum used can also be a point such as reflective dot 1922 
permanently attached to the screen itself, preferably near the corners or other locations 
which are supported. This too will move inward in Z when the finger touches the 
screen. 

[00726] It is desirable to have a portion of the instrument panel such as 1 975 
extend past the screen 1901 as shown in order to shadow the datum's from sunlight 
such as 1978 coming from inside the passenger compartment . Alternatively a surface 
of the screen can be blackened in the region to blo'ck the sun. 

[00727] The support can be of a plastic or rubber or other such compound which is 
compressible, rather than elastic members such as steel springs. This is much less 
costly, but creates a more difficult solution for touch location due a less predictable 
response to touch forces, which further may not be purely in the z direction . 
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[00728] For solution of touch presence, one only need detect that any of the 4 
corners has moved from a previous position. The normal untouched position (in the 
absence of vibrational effects) can be detected on car startup or any other time desired. 
One can sum the movements of all corners as well. And one can additionally look at 
other datum's internal such as locations of knobs, switches and the like, since they too 
will move under the force of touch if secured to a screen which moves in a rigid body 
manner. 

[00729] If touch presence is solved for, this can be useful if only one touch icon 
box is projected for example. And, as pointed out in previous applications, if one 
knows a touch has occurred, one can use vision techniques disclosed elsewhere herein 
and in other cases to see the finger location directly. Such touch "Pre-sensing" using 
force is useful to avoid touch signals which can possibly be falsely generated when 
ones finger or objects are close to the screen but not touching. 

[00730] For actually solving touch location, this requires in general analysis of all 4 
signals from the support members, and possibly other points as well internal to the 
surface, in order to determine where the touch has been. For example in fig 19a, the 
touch of a radio button is determined to be, for example, 3 inches from the location of 
supported edge 1905, and 7 inches from support 1906. If all the touch buttons to be 
pressed are in a horizontal line such as dotted line buttons (e.g. radio buttons) 1980- 
1983 extending from one side to the other, then it could be expected that button 1980 
had been selected, since movement of datum relative to 1905 was proportionately more 
than that relative to 1906 

[00731] It should also be noted that the camera can determine the location of 
datum's relative to a housing that surrounds the screen , rather than to itself. This 
allows the camera to vibrates slightly without affecting measurements of displacement. 
For example not only could datum 1960 in figure 19b be determined as to position, but 
also the datum 1990 on fixed member secured to the instrument panel nearby, with the 
distance between them in the camera image solved to provide information as to 
displacement. This value changes due to the angulation of the camera in this case . 
Such a fixed member could be a support ring continuing around the whole 
circumference of the screen for example. 
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[00732] If the screen does not move as a rigid body under touch action, then 
solution of the position equations is more difficult. But typically in many applications, 
the degree of accuracy needed to discern what is needed is low (e.g. the choice of 4 
radio buttons above). And too, the camera by virtue of its whole field measuring 
capability allows one to see intermediate parts in the screen surface if needed to aid the 
solution, for example added datum 1971 in figure 19a which can be put there 
particularly to aid such solution (and for example could even be a fixed point on a knob 
holding member of knob 1 91 1 which would be there for another reason, and which 
would also serve to shadow the point. 

[00733] It should also be noted that one can calibrate a screen and its mounting 
for its deflection characteristics once built . One can just touch the screen with known 
touch type forces at all the points one would like to sense touch (or z axis knob location, 
or z axis switch location) at, or at a representative grid of points,, and determine the 
response in terms of displacement. By storing in computer memory the values of 
displacement obtained, also perhaps at multiple points across the screen surface, in 
addition to the points supported, one can correlate any future touch to these values. 
This is particularly effective since one also in many applications has control over where 
on the screen the touch point icon boxes one might use (e.g. the radio buttons in fig 19) 
are to be projected . It is often not needed to have a table of more than 50 positions 
where displacement vs. force is calibrated. Some systems can make do with 10 or 
less. And one can just assume in many cases a unitary value of force for a typical 
persons touch. This is not costly in terms of either processing time, or calibration and 
test in factory. It alternatively can be done in situ after the car is manufactured. 
[00734] This can be done too with neural nets as pointed out elsewhere in other 
contexts. 

[00735] It is noted that the calibration of the sjcreen using a digital matrix array of 
photo detectors as the sensor, avoids analog calibration drift experienced in many touch 
screen designs, while still allowing high resolution. 
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Figure 20 

[00736] Figure 20 illustrates an alternative display device employing a flying spot 
scanner and laser or LED sources and the additional use of such scanners to perform 
sensing of both control details and fingers or other touching objects. 
[00737] As shown in the figure, a Microvision brand MEMS based xy mirror 
scanner 2001 driven by drive controller 2002 controlled by computer 2005, is driven so 
as to sweep a focused beam 2015 from lens 2006 combined from one or more diode 
lasers raster fashion across screen /control surface of the invention 2025. In one 
example of such a flying spot scanner, 4 Diode laser beams of different colors are 
combined by combiner 2030 using the apparatus of fig 15b, dichroic mirrors, or other 
suitable means known in the art. Three of the lasers colors are typically are red green 
and blue, in order to allow multiple colors to be generated by appropriate combination of 
intensities from each laser. The 4 th is purposely chosen to be outside the visible 
wavelengths (e.g. IR at 880nm) in order to act as an interrogation beam for finger and 
knob position which can be energized independent of projection color and unseen by 
the driver. 

[00738] The reflected IR light 2026 from objects such as a knob or ones finger 
2040 in contact with screen surface 2025 is sensed by detector 2050 whose associated 
lens 2051 aperture is such that light only from a region around the point being swept is 
accepted, since the lens also looks down the same beam path by virtue of beam splitter 
2052. Narrow band Interference filter 2053 accepts only the IR laser wavelength at 
880nm in this case, and rejects virtually everything else. 
[00739] Signal processing in the simplest cases looks for abrupt changes in 
reflectance, either due to the presence of fingers, or from datum's on knobs and other 
physical details of the type disclosed above and in copending applications. Such abrupt 
change detection (e.g. using a high frequency AC filter ) coupled with the wavelength 
filter, effectively eliminates slowly changing sunlight and other optical noise. The useof 
a single detector with the flying spot device, also allows one to modulate the light source 
and correspondingly demodulate the detection at a very high frequency which can 
further discriminate against background light. It is noted one can also use projector 
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light for the sensing purpose, without the added IR source. However, this requires 
coordination of projection with sensing which otherwise is not required. 
[00740] This arrangement is thought to be the most attractive from a cost and 
optical noise rejection point of view. The September 2002 issue of Worth magazine 
indicates that Microvision Corp. believes the RGB version could cost only $40 in high 
volume, of a size capable of replacing a conventional TV set. This includes the lasers, 
optics and scanner, and maybe the scanner driver. Thus the version herein, even with 
the optional IR laser and detector would cost only $50 lets say, at which price the whole 
RTD based instrument panel becomes extremely cost competitive with all known 
alternatives. 

[00741] It should be noted that a flying spot type scanning projector can be used 
even if the sensing is done with a camera as disclosed elsewhere. Conversely, such a 
detector equipped scanner can be used in any of the embodiments above as a sensing 
device only, in place of the camera. 

[00742] This scanner can also be used for another form of optical touch sensing , 
based on light reflected from a member deformed by the finger touch. As can be 
appreciated as shown in fig 20b, when an outer screen member 2070 is deflected even 
slightly by a finger 2075, the light beam coming from scanner 2071 is reflected as ray 
2072 in a quite different direction than it was 2073 (dotted lines) from the undisturbed 
surface. By comparing the return signal of an initial no touch state, to that with touch a 
determination of touch can be made. In the case where the distortion tends to increase 
the signal, this adds to any signal from the direct reflection from the finger as well. 
[00743] The sensing ability can be operational while projection of images is taking 
place, or alternatively performed during dwell times when it is not. 
[00744] Another discussion of a related embodiment may be found in my 
copending application 09/568,554. 

[00745] It should also be noted that the use of logarithmic or other sensitivities in 
sensors used to scan the surface of the screen in a flying spot manner can be used as 
well. However most single photodiodes can be made to operate with very high dynamic 
ranges so this may not be necessary, even where sun load is high. Detector output too 
can be used to gain control the projection sources (eg semiconducting lasers or leds), in 
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order to put more energy on the screen when viewing conditions are difficult due to high 
ambient light . 

Figure 21 

[00746] Figure 21 illustrates embodiments for control of sensing and projection , 
and image processing steps, including methods for determining finger touch and 
physical detail location also in the presence of significant and variant background 
illumination. 

[00747] There are basically two issues. First is to see the correct data with the 
camera or other detector, in the presence of what can be strong background coming 
from inside the passenger compartment. This situation is discussed further in fig 22 
below. 

[00748] The second is to identify from this data the physical detail position or 
finger location, or the movement of either . This can be done using relatively simple 
machine vision processing algorithms, such as image subtraction, blob analysis, edge 
transforms and the like. For the physical details this is made much easier as the 
knob/slider or switch datum's can be chosen to be of good or excellent contrast, and 
further can be made in easily recognized shapes, or colors (if colored light source is 
used) . In addition they in locations known apriori to the system, such that one can look 
for the desired characteristic(s) in that location. For example in a certain annular ring, 
three targets on a knob, or alternatively one radial rectangular target on a knob would 
have to exist. 

[00749] In the case of the fingers it is more complex and this is the subject of 
several figures below. But in the simplest case, and in the absence of strong 
background, one again is just looking for a round or oblong shaped blob in a certain 
range of sizes (typically just defined as an pixel area of blob) represented by the finger 
in contact with the screen The reflection from flesh in contact with the screen works 
well in this way, though using IR led sources at 940nm I have found too that gloves, 
even many black gloves can also be seen . 

[00750] As pointed out elsewhere and in co pending cases it may be useful to 
subtract the background with or without the illumination source on from the instant data. 
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Because this can take longer, a higher frame rate than 30 frames per second typical of 
TV cameras is desirable for best results. This can be achieved locally with ease (e.g. in 
the region of a knob) using a pixel addressing CMOS camera for example, which can 
easily provide data to the computer to allow it to do several hundred alternate 
background subtracts per second 

[00751] Shown in figure 21 are processing steps to determine physical detail 
location. In this first example, the projector is the light source for sensing as well. The 
projector 21 01 is fed data by computer 21 05 to cause it to display on the screen 2110 
whatever image is desired for viewing (which could be a blank image as well). In the 
back region of knob 21 15 an annular ring of light from the projector illuminates the 
annular ring of the knob where datum's lie. In this case only one datum 21 17 is shown, 
a radial line shaped one, but more datum's and other shapes can be used as desired to 
suit the accuracy requirements needed . The datum's can lie on the back of the knob 
on the driver side of the screen as shown above , which allows the knob to be 
transparent in its center . Or as shown in this figure the screen can be cut out to form 
hole 2130 in the center with the actual diffusing portion of the knob 2135 located on the 
knob itself, which rotates in a bearing 2140 affixed to the screen with adhesive or other 
means. In this case the datum's, such as line target 21 1 7 are rotating radially like a 
clock hand. 

[00752] Camera 2150 images the reflection back from the datum 21 17 to obtain an 
image of the datum which is analyzed by computer 2155 to determine the 
circumferential location of datum and thence the knob, and thus the selection or value 
desired by the user. This datum may be in reference to optional targets on the corners 
of the screen or elsewhere used to register location, independent of camera position 
and vibration if present. 

[00753] It should be noted that since measurement can be made very quickly 
using suitable cameras, that the projector need only be on for a short time, e.g. 10, 



msec. 



[00754] For the sensing of finger locations, generally speaking the touch can be 
anywhere allowed by the projection program (typically indicating to the person where to 
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touch) or alternatively in areas preprinted on the screen or on an overlay placed on the 
screen. These latter two alternatives are less generic and less likely to be used. 
[00755] The touch location can be either of an icon box, or on a indent or bump 
on the screen, or more generally a random gesture such as sliding ones finger along a 
line of action, possibly guided by an indent or raised ridge on the screen. 
[00756] For the generic case, one must be looking anywhere a touch can be 
registered and deciphering images obtained of the total available region in order to find 
the touch location. This can be either by directly looking at fingers, or alternatively by 
looking at something the finger does to cause variation in the electro-optical signal 
received, either in the intensity of reflected light, or in the position of imaged datum's. 
Both of these alternatives will be discussed further below, and have been treated to a 
degree in other copending applications. 

[00757] For the direct viewing case, there are several possibilities, some of which 
have been discussed above. One can look for example for the characteristic shadow 
produced by the finger when illuminated from behind using light from the users side. 
Alternatively, and or in addition, one can search the camera image for the characteristic 
round or somewhat round blob forming a closed polygon when illuminated from the 
projector or an independent light source such as a LED from the rear. This 
characteristics can be assumed or in some cases taught, by just asking the driver to 
touch the screen in his unusual manner in a certain areas, and memorizing in the 
computer 120 say, the image signature . This too can act as a theft prevention mode 
too by going through this routine each time the car is started. 

[00758] One can also train a computer based camera system , for example at the 
factory, said system having a neural net with different sized drivers fingers and colors of 
fingers so that it can be taught to recognize all reasonably possible variants. Such 
training as well could be for all possible positions of knobs sliders jand so forth as well, . 
and in the case of different lighting, such as with sunlight effects . I 
[00759] The knob targets are typically bright on a black background, but they 
could be reversed. In this way (dark on bright background) they would not be confused 
by stray light brightness zones caused by large sunlight loads 
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[00760] It should be noted that the camera sensor system of the invention can 
identify touch on the screen, or a short distance from the screen, if sensitivity is 
increased. One thus can operate the system in two modes, only touch (also the 
occurrence of which can be verified with force or acceleration sensing), or with gestures 
which are directly in front of the screen, without necessarily touching. This also allows 
one to place an overlay on top of the screen and view touch icons on it, by looking 
through the overlay. 

[00761] This later situation is useful where one might for example wish to have a 
large screen with no knobs or other permanent physical details on it, which could be 
used as a large touch screen . For example a 16x9 inch screen in a Military "Humvee" 
Vehicle, used for display of a Tactical battlefield map. Then when desired, you can 
place an overlay over the screen, if desired itself large enough to cover the whole 
screen area, and by seeing through the permanent screen, identify the position of knobs 
, sliders, switches or fingers, for example on the overlay . This overlay for example, 
might be specific to a certain Humvee Variant, or for example an overlay created, with 
its accompanying software, for a particular mission. Alternatively, certain parts of the 
overlay could be cutout to allow the base touch screen beneath to be contacted. 

Figure 22 

[00762] Figure 22 illustrates further processing aspects relating to problem 
sensing areas, particularly lighting. 

[00763] In a vehicle application, there is a particularly large variation in ambient 
lighting conditions in the passenger compartment to which the screen of the invention 
interfaces. This variation ranges from dead black at night, with no lights on in the car or 
in the vicinity, to bright sun pouring down thru the windshield, or a sunroof, directly on 
the screen. This situation can also occur with sun coming in from the side at certain 

I 

times. | 

[00764] The question then is, what does it take to operate over this range? And 
secondarily for the bright direct sun case, is it required to so operate, since such lighting 
will wash out virtually any information displayed anyway, as it does today on displays 
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which are in the open (and not hooded as most instalment clusters are for this reason). 
Prismatic films such as made by 3M optical products division can help with this. 
[00765] I believe it is generally essential that the unit operate in all cases but the 
washout one, and even then I feel it should sense the physical details such as knobs, 
which then could be used for critical items to which washout of data was not an issue. In 
some cases this could require printing of characters on the screen (or an overlay) in 
addition of projected characters if it was necessary to always read something. In other 
cases the system optionally could, upon sensing via the camera of a super bright 
condition in the region of a certain knob (or the whole screen) switch over to a voice 
description of knob position for example if the actual the projected knob lettering could 
not be seen. Or the light energy projected can be increased in the problem region to 
allow visibility in sunlight condition. 

[00766] In work to date, the invention in all embodiments has functioned well for 
sensing a night or in modest daylight conditions. And it has functioned in all cases for 
the sensing of physical details. This is largely because the datum's on the knobs for 
example, are shadowed by the knob itself. 

[00767] But what about the fingers? Clearly direct bright sun can overpower the 
effect of any light source with in the unit. The answer to this is singularly or in 
combination to 

[00768] • Use the peculiar shadow effect of the finger contact area; 
[00769] • Use the shadow outline of the finger; 

[00770] • Use specialized cameras or other detectors whose integration time 
or other sensitivity can accommodate the very wide dynamic range required; 
[00771] • Use a deflection based system which does not depend on light 

intensity for the answer; 

[00772] • Use another form of touch screen entirely, arj appendage so to 
speak. This is always an option but is not Tree" like the optical opes (assuming the 
optical is used for knobs and such), and thus is not appealing for high volume use. A 
high grade touch screen of the capacitive field type can add $100 in volume. 
[00773] Assuming as in the case of touch icon box actuation that we know the 
regions of interest where to look for touch apriori. In this case, we can modify camera 
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function locally to optimize exposure for each region independently., and it should be 
noted this can be done in a general way, even if we don't know where to look a priori. 
One can also utilize a logarithmic response, or lin-log response CMOS or other camera 
(e.g. the FUGA 15 one also mentioned in a copending application) for this purpose, 
which is not bloomed or otherwise made unreliable, or inoperable by massive sun load. 
[00774] I have found that by comparing the finger with the projection light on, to 
the projection light off, but shadowed, that a good answer can be found in almost all 
cases but enormous load. I have also found that just looking at the finger in the high 
sun load condition, without reliance on the projection light at all (From auxiliary IR 
source or the data projector ) can also be used. Consider fig 22a, where finger 2200 is 
touching screen 2201 of the invention in a typical manner indicative of a push of a 
button, such as virtual radio presets . The camera 2220 observes the back side of the 
screen, and the image 2260 obtained is processed by computer 2250. In this case, the 
finger is flattened a bit at the end 2205 pressed by the person into contact with the 
screen, and this flattened area effectively shadows the sun light 2210, even though 
other portions of the screen are extremely bright. As a result shadow region 2215 in 
the camera image 2220 can be sensed and compared to the bright region 2225 
around it . 

[00775] In the case where we know where the shadow can be to make selections 
(E.g. on a projected preset) we can just look in those regions (e.g. 2221 and 2222 in 
image 2260), and if we have sensed a bright sun condition for example by looking at the 
sum of camera pixels in the image, if a shadow (i.e. a region of dark in a sea of bright) 
roughly round or somewhat oblong is there, like 2215, then one can assume that's a 
finger and determine if it is touching the button location in question. In this case it is 
determined that it is touching in the region of projected button 2221 
[00776] Another way to do this is to sense the edges of the shadowed finger as 
shown in copending applications, and use the edge shape to predict where the finger is. 
This can make a prediction of location of the tip of the finger, even without the hard 
contact needed for the black shadow condition above. In this case with the edges off 
the screen, the finger shadow is grayer but still very distinguishable as a finger. This is 
also because the typical car of today has nobody in the front middle passenger position, 
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and the drivers finger has to approach a screen in the center stack from a range of 
known angles from the drivers side. And thus the range of shadows in this manner is 
relatively limited. However, the variability of this is still a problem. 
[00777] It should be noted that when a specific shadow condition is indicative of a 
finger in one instance, and a bright spot is indicative in another, then one can effectively 
solve for both, and if either is present, one can determine finger presence. And 
certainly if both are in the same place (this can occur when the finger reflects projected 
light from the region being shadowed due to finger contact, but where the background 
light is intense enough to shadow the remaining part of the finger around this zone, 
which does not reflect sufficiently as it is not in contact with the screen. 
[00778] A shadow can be caused on the screen by something other than a finger, 
so it is important to look for shadows that have finger like shape and/or reflection 
characteristics . This also includes the unusual effects around the edge of the finger 
when strongly illuminated from behind which creates a gray boundary apparently due to 
light leakage around the finger and through the skin at the edge of the finger. 
[00779] It is also or additionally possible to determine the general outline of this 
gray region and predict where the fingertip is, as has been described in copending 
applications. A Sobel transform of the image, for example, gives the outline in many 
cases of the total finger extension, particularly when the projection device (E.g. LEDS) 
is turned off such that no projected light returns from the finger. This technique can act 
as a check on a finger location determined from the black shadow indication or the 
projected light reflection or the comparison of those two. Other image transforms can 
also be used such as Houghs. 

[00780] Another method of discriminating is to use a monochromatic wavelength 
of illumination, such as an IR LED or laser and to bandpass filter the return to the 
camera. This has been described in copending apps and previous embodiments herein. 
With the LED sources of choice this can by itself discriminate against most conditions 
other than direct sun. With laser sources, the band pass discrimination can be even 
better using narrow band interference filters. 

[00781] Another method is to modulate the light source at a high frequency and 
demodulate the detection. With CMOS and other such TV cameras this can be 
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achieved to a degree by electronically shuttering the camera , leaving it open, only 
when illumination light (e.g. from an IR LED) is projected 

[00782] Another method is to compare touch at different locations. For example, 
assume there are 5 radio buttons more or less in the same region of the screen. It is 
known logically that only one of the 5 will be touched and the other 4 purposely not 
touched. By cross comparison of light from the area of the buttons (no matter on what 
principle is used) one can find the one (touched)whose signal that does not correlate 
with the others (untouched) . 

[00783] Another method is to look for movement due to the touch , for example as 
one uses ones finger as a slider, moving across the screen. This is more difficult as a 
tracking function is required. But if one takes "N" readings of data, one can determine a 
trend in the data (e.g. movement from right to left on the screen) . In many instances, 
even if a particular reading cannot be obtained, the function can be satisfactorily 
provided to the user (e.g. heat increased). 

[00784] it should be noted too that the reflectors on the knobs/sliders switches if 
used can be retroreflecting comer cubes, or scotchlight material or alternatively diffuse 
reflective material, to suit the characteristics of lighting used to illuminate them. 
[00785] To recap, for bright sun conditions, all knobs and other physical devices 
can typically be constructed in such a way that their very shape and location shadows 
light from the sun or other sources in the passenger compartment. Thus for those items, 
the projected light is what is typically used to sense, in any day or night condition. 
[00786] For the case of fingers sensed by machine vision processing of direct 
camera images, then for sunlit conditions, it is useful to look for the black shadow of the 
contacted finger and compare that to the projected reflection image at the same location 
- which is thus shadowed. This answer can be further compared to the finger edge 
outline condition, which if present confirms the finger location. We note however, that 
sometimes other fingers can also be seen under such shadow conditions,: so the edge 
outline case may not by itself be satisfactory. 

[00787] To guard against extraneous indications of touch, which could occur in 
unusual lighting conditions or finger or hand movements in front of the screen, it should 
be noted that as disclosed in previous copending applications one can combine a force 
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sensor or other sensor with the invention in order to tell that a touch has occurred. For 
example accelerometer 2252 located on screen 2201 can be used to sense that the 
finger has hit the screen (with a force significant enough to register a momentary 
detected wave in the screen). This signal can be used to tell the camera to read, and if 
more sophisticated processing is used, from multiple accelerometers, one can even 
predict the location of touch. 

[00788] Acceleration may not be best for light touches which produce little shock 
to the screen member. In this case other alternative sensors can be used to indicate 
touch, for example resistive screen elements. 

[00789] It should also be noted that for extreme sun conditions, when the screen 
washes out at least in ones ability to view it satisfactorily, that the invention 
comprehends sensing this condition, and increasing the projection or other display 
brightness or contrast. In addition, the display can change color, even to the point of 
switching to a negative image of what was previously displayed, if that aids in 
distinguishing data in sunlit conditions. 

[00790] This change in brightness, contrast or color can also be performed 
selectively in those regions washed out, leaving the others unaltered, or not as much 
changed, to leave a degree of continuity with previous information. This is also true 
because in many cases the sun problem is fleeting, and adequate display conditions are 
restored after a short while. 

Figure 23 

[00791] Figure 23 illustrates an alternative instrument panel embodiment having a 
camera external to the display which follows from the disclosures of the co-pending 
applications incorporated by reference herein. 

[00792] As shown in figure 23 a camera 2300 connected to computer 2301 is 
located in the headliner or overhead housing of a vehicle roof 23Ci5 and positioned so 
as to observe the instrument panel. The field of observation can be quite large but is 
limited in this example to the region of the center stack 2310, and the control display of 
the invention therein, (other co-pending applications have disclosed such cameras 
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observing switch gear and hand gestures for example on the steering wheel and 
transmission lever area). 

[00793] The use of such a camera type application has several appealing aspects. 
First it can look at a lot of different data, not just knobs, switches or fingers, but also the 
location of other parts of persons, and indeed other persons entirely. This can be used 
to detect for example, some ones head nodding off to sleep. And hand gestures in the 
passenger compartment can also be sensed 

[00794] While discussed here relative to the center stack in the front of the vehicle 
such an approach also allows one to provide added functions to rear seat passengers, 
without running wires or power. In short the camera image (or the similar flying spot 
generated image of figurer 20) can be used to provide sensing at many points at once. 
This is useful for example if one has a DVD player overhead, and you want each person 
in 4 rear seats in a minivan to be able to operate it . Today, to do this safely (that is with 
each person buckled up) one would have to have two and possibly 4 control panels. All 
this costs a lot of money - and precious space. With the invention of this embodiment 
(and copending applications) a single camera and computer vision processor can do it, 
and the processing may be time shared with the control aspects of the RTD, saving 
cost. 

[00795] To operate a camera one needs IR LED or other light sources, preferably 
invisible to illuminate the areas to be sensed when daylight conditions or other suitable 
lighting is not present. This can be easily provided by locating such sources typically in 
the headliner, for example even inside the housings of normal lighting of the vehicle. 
E.g. in a dome light. 

[00796] The camera and computer in this embodiment example may accomplish 
several things. First it determines the location of physical details such as knob 2320 and 
slider 2325. Second it can determine the location of a finger such as 2255 touching the 
screen. This is particularly interesting here in that if the camera operates at a high 
enough rate, it sees the approach to the screen as; well. 

[00797] Third it can see a hand or other gesture. In other words one can signal 
the computer without touching the screen at all. For example one can determine the 
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pointing direction of ones finger pointing at the screen or something in the vehicle which 
needs to be controlled. See also co-pending applications. 
[00798] Another useful gesture is to nod ones head in certain manner, or move 
ones finger - for example when it positioned on the gear lever . Fourth it can see other 
things not on the screen. 

[00799] The knob or finger location, or other datum location, can be determined 
relative to a previously determined position, or absolutely relative the cameras image 
field. Or as is often desirable, relative to some other point or points nearby on the 
screen which are fixed, such as trim indicia 2330 shown. Even if the camera should 
move a small amount, as it might due to vibration, the relative data is available in such 
manner as to determine the desired input from the relative location. 
[00800] For directly viewed body gestures there is the possibility of having false 
signals. For this reason it may be desirable to only register such gestures when some 
other situation is present. For example ones hand on a gear lever. The finger can 
actually be a shift cue, like a paddle shifter in a race car. If you Flick your finger up it can 
be sensed by the image analysis software and the computer can tell the car to shift . 

Figure 24 

[00801] As can be appreciated, the machine vision based touch sensing disclosed 
in fig 21 and 22 above can function with both hard and soft screen surfaces, as it 
depends on the image or other optical signal of the person's finger itself. And at the 
same time it can see the knobs and other physical details required for control. 
[00802] Fig 24 illustrates a touch sensing embodiment of the invention that utilizes 
deflection of the screen. This deflection can be measured in a number of ways, such 
as those of USP 6008800 by the inventor , also using if desired a computer camera 
based system which can see the knobs sliders and other physical controls as well. And 
a deflecting screen can have other functions providing tactile feedback by selectively 
expanding all or portions of the screen surface too as disclosed as well in referenced 
applications. 

[00803] The screen outer member deflection under touch which is needed to be 
detected can be quite small (e.g. .010 inches , though this depends on the sensing 
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method used), as is desirable I believe , i.e. that it not have too much "give" that the 
user feels it is too soft. It is however, noted that a soft deflecting surface may be safer 
in case of an accident than a harder one, and thus preferred at least to a degree. 
[00804] We also note that it does not have to deflect in regions not touched. If 
touch activity is all in a certain area, the screen surface can be rigid elsewhere, and 
thus can be supported more easily. 

[00805] As illustrated in figure 24a , a screen 2400 is supported on its edges 2405 
and 2406 (other edges not shown in this sectional view). As finger 2410 presses in the 
screen deflects around the finger and at all points between the supports, with the 
deflection at each point a function, for example, of the screen material, the support 
location and the cross sectional shape of the screen. Solution of the deflection 
equations can allow the finger location to be found in x and y, And if more involved 
calculation is undertaken, and enough data points are available, in z as well (which is 
thus related to the force of finger touch, since the more force, the more z deflection). 
[00806] For simple systems, where deflection is expected in widely spaced 
regions, one can simply use the camera system to solve for maximum deflection, and 
where this is determined to exist, the touch is assumed more or less to be. For example 
in fig 24a camera 2415 observes a grid pattern of points such as 2420-2424 projected 
on the screen 2400 by ir laser 2430 whose beam 2431 is split into a pattern by grating 
2435. This pattern and the camera field of view can extend, and normally would extend 
in both axes of the screen. 

[00807] As the touch is made, the force of touch of finger 2401 acting on the 
screen 2400 supported at its edges causes the point P to move inward and this moves 
the projected point 2422 on a surface of the screen (in this case the rear surface 2441 ) 
more than the other points, which is determined by camera 2415 whose optical axis 
2445 is spaced in angle from that of the laser and grating orders (and a computer 
associated therewith not shown) using for example triangulation processing contained in 
US patents by the inventor and his colleagues Alternative to projecting the points 2420- 
2424, one can have these points permanently a part of the screen, for example such as 
white mark 2440 on the back of the screen , which if a beaded 3M type blackened 
screen could be in a region where the glass beads were not present 
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[00808] Alternatively the glass beads themselves can be used as such datum's, 
due to their reflection (and transmission, in the case of sun illumination from the 
passenger compartment). One can look at a single bead, but this requires high camera 
resolution (beads are typically only 0.002 inches in diameter). Easier is to correlate the 
movements of a group of beads, which effectively move together as the screen is 
deflected. 

[00809] Fig 24b illustrates a touch sensing embodiment of the invention that 
utilizes a relatively soft outer screen material such as disclosed in copending 
applications such as 09/568,554. This too is used with 3M beaded material which to 
function properly requires an air interface on the projection side. 
[0081 0] As shown there is a rigid transparent backing member 2450 which 
supports a 3M screen material 2451 having beads on its blackened back side 2455, this 
screen material is further laminated to relatively softer plastic outside member 2456. 
The screen 2451 is spaced away from member 2450 by small spacers such as 2460 
which are arrayed at suitable intervals. These spacers are small in diameter (E.g. a few 
thousandths of an inch and can be placed as needed, also considering the projection 
device). Spacing of beaded member from the rigid support member can also be 
achieved by air pressure, in addition or alternative to the spacers. 
[0081 1] As finger 2470 presses in on the outer member 2460, it is deflected and 
eventually forced against the rigid member. This makes the projection not disperse 
properly at that point but doesn't matter as covered by the finger and cant be seen 
anyway. As disclosed above, camera 2480 can observe points projected onto the 
deflecting members or actually provided on them, for example datum's 2485 and 2486 
whose movement under the force of touch allows one to predict finger location, since 
datum 2485 was observed for example to move .020 inches, while 2486 moved .010 
inches or less given that the space was between it an the point of finger touch in one 
direction at least, meaning the finger touch was closer to datum 2485. This solution can 
be made in the other plane as well, where xy grids of datum's are employed (as is 
usually the case). 

[00812] This soft screen does not impair knob or other physical detail function. A 
knob (or other physical detail) 2488 can be attached to the screen outer surface in an 
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area where it is more rigid, or can be secured to rigid backing member 2450, as shown 
using a pin and post arrangement 2489. 

[0081 3] It should be noted that moire grids can be used to optically process grid 
data to determine changes due to touch by providing a reference. However one can 
also just remember in the computer memory where the previous points were located, 
and look for change in their position. 

[00814] In the same vein one can calibrate a screen for its deflection 
characteristics once built. For example, one may indent the screen with known forces 
at a representative grid of spaced points, or if known apriori, at all the points one would 
like to sense touch (or z axis knob location, or z axis switch location) at, and determine 
the response in terms of deflection. By storing in computer memory the values, also at 
multiple points across.the screen surface, one can correlate any future touch to these 
values. This is particularly effective since one also in many applications has control 
over where on the screen the touch point icon boxes one might use are to be projected. 
It is often not needed to have a table of more than 50 force positions where deflection is 
known and calibrated. Some systems can make do with 10 or less. This is not costly in 
terms of either processing time, or calibration and test in factory. It alternatively can be 
done in situ after the car is manufactured as well. 

[0081 5] The calibration can also be specific to what is desired to see. For 
example if all that was desired was to determine which of four radio buttons across a 
screen were pushed, this could be done by just calibrating the deflection seen at 
different points on the screen, when a typical contact force was exerted at those points 
in succession. The data taken is then used to correlate future events to those touch 
indications, when radio buttons are projected in those locations, (or in general). The 
deflection is compared at each of the locations and to the known ratio of deflection at 
the time of calibration. 

[00816] Since the spacer such as 2460 separate the two members, it should also 
be noted that these members on their adjacent faces can further incorporate resistor 
elements to register presence of a touch signal, or the location of touch, the latter in a 
manner similar to conventional touch screens of the resistive type. 
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[00817] For example consider optional transparent electrodes on adjacent faces of 
members 2450 and 2451 respectively , connected so as to register a current flow in 
meter 2490 if they are in contact when a voltage V is applied. 
[00818] In operation ,one can purposely not utilize camera 2480 to sense touch 
until computer detects some minimum level of current flow in the circuit between the 
transparent electrodes. If presence of contact is all that is required, the electrodes can 
be relatively far a part on the surface of the screen, since the accuracy of location is not 
solved by the resistive device. A simple matrix type of resistive touch screen 
technology known in the art can be used in this case, particularly since one has in many 
instances apriori knowledge of where on the screen the user is to touch (i.e. where 
projected data occurs). 

[00819] In this mode and others described herein, When a force or current or other 
touch condition is detected, the system is programmed to look, minimizing the chance of 
false signals due to unusual lighting conditions. It is also noted that one or more 
modulated break beam type LED and detector pairs external to the screen, such as 
2360 and 2361 shown in dotted lines can be used to detect the touch presence, as can 
cameras external such as also described in fig 23. 

[00820] Discussed now is an embodiment utilizing an organic light emitting diode 
(OLED) display, which is self emitting and has a more compact layout than that of a rear 
projection versions, while still having many useful features thereof. 

Figure 25 

[00821] Figure 25 illustrates an alternative OLED based display device employing 
camera based physical detail sensing of the invention and touch sensing also using 
deformable screen based touch capability described in the referenced patents and 
applications . 

[00822] Shown in fig 25 an OLED Display 2501 , for example of the type described 
in USP 5739545 by Guha et al, including a backing member 2502 mounted in 
automobile instrument panel 2503. In this case the pixel elements of the display 2501 
are self luminous and no back lighting is required, as it is with LCD flat panel displays. 
In addition, many papers predict that such OLED displays will also be able to be 
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economically made curvilinear, rather than flat panels, a very desirable feature for 
instrument panel use, especially when large in size so as to conform stylistically with the 
rest of the vehicle. 

[00823] To utilize this type of display, the form described by Guha with a 
transparent substrate and at least partly transparent cathodes and anode structures is 
desirable as it allows a degree of transparency to certain wavelengths such as that of 
light source 2504 on the rear side, where also is located camera 2510 which in this 
invention is used to sense control detail positions and persons touch locations. To 
avoid problems with human vision seeing the source, a desirable wavelength of source 
2504 is in the near infrared, for example at 880nm. But it is recognized that the choice 
of illumination wavelength also may depend on the construction of the OLED device 
itself. 

[00824] It should be noted that one can also light the datum's on the knob, or the 
finger touch, using the light from the OLED elements themselves, assuming such light 
can be retransmitted back thru the OLED display to the camera or other sensing device 
used. For example a group of OLED pixels 251 1 can be used to illuminate datum 2550, 
in addition or alternative to the use of light source 2504. 

[00825] In this example, the camera 2510 is used to see both datum's such as 
2550 on the rear of knob 2520 or the finger 2525 touching the OLED display front 
surface 2526 (or an overlay 2527 thereon). The knob is secured to the display by 
appropriate means, and a bearing 2551 it rotates on or in can be bonded to the front of 
the display or overlay. Alternatively, a pin through the substrate 2531 of the display 
may be used to secure the knob, and if desired transmit rotation of the knob to a datum 
on the rear of the substrate which rotates with the knob and can be sensed by the 
camera, as taught above. 

[00826] As noted elsewhere, in this example the determination of touch and touch 
location is by direct view of the finger. However, to effect this it is not necessary to see 
the finger in total. For example, due to the OLED structure, there may be circuit regions 
which are not transparent which prohibit one from seeing the finger (or physical detail 
such as a knob, if datum's are located on the drivers side, rather than transferred for 
example by the pin to a rotating member on the rear of the display such as taught in 
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figures above) at all points . However enough points can generally be seen, such that a 
sufficient estimate of finger location or knob position can still be made. As one 
processing example, a finger shape and size template can be stored in the computer 
and sequentially matched to the instant image, to find the point of best match and thus 
the likely finger location 

[00827] Another way of optically sensing finger location on the OLED display 
shown is to determine deflection of the display when finger force is applied. This has 
been shown in co-pending applications and other embodiments herein. Alternatively, 
the whole screen force based displacement can be solved as noted above to obtain 
touch location. 

[00828] The invention herein can be used with inorganic electro luminescent 
displays as well as organic ones. 

[00829] It should also be noted that LED sources can be utilized with a normal 
white light projector to be used as a back up to a white light bulb, so that in the case of 
bulb failure, the LED or LEDs can be used to provide usable screen illumination if need 
be. The LED can be on all the time as a supplemental source, or energized when bulb 
burn out is detected. And it may not be necessary in this case to have all colors 
displayed, thus LEDs with particularly powerful and visible wavelengths may be used, 
with out regard for having a beautiful white balance. To a degree this is true in all 
cases, if one does not wish to represent true color images on the screen. The later is 
desirable surely, but not needed for control or other vehicle operational purposes. 
[00830] Another method of helping keep things going in case of bulb failure is to 
illuminate only key portions of the RTD screen with the LED or LEDs. 
[00831] Finally, another example is the case of an RTD control using virtual 
controls on the touch screen portion, and physical controls (knobs, dials, slidecs, 
switches, any or all) on another portion (which also pay have touch capability too). 
Typically the virtual portion would be the top portion, the physical thelpwer, but hot f - > 
necessarily. The reason for this would be to put the expansive image Portion capable of 
providing backup and other images in the line of sight of the driver. This portion would 
not be cluttered with physical devices, generally speaking. And key controls could be a 
bit lower and out of main the sunlight load (see below). 
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[00832] It is desirable that the most critical controls be physical, since they are 
easiest for people to understand today, and importantly, they can be easily grasped and 
worked, even if the total control space is brightly illuminated by sunlight which can tend 
to wipe out both the displayed image and cause potential trouble for some kinds of 
touch sensing. One can also used the sensed data from the camera of the invention to 
control the illumination power to increase same where sunlight is tending to wash the 
display out. In some cases this could mean changing colors of the display as well in 
order to produce colors such as green which are more readily viewable by the person, 
assuming that the projector itself can produce green as efficiently as red , for example. 
[00833] It is desirable to make the display optimally bright in critical areas where 
control functions need to be executed. In is not necessary on a large area display 
having video image areas not used at a given time for control purposes, to necessarily 
increase all areas at once. Thus in some cases such as scanned laser projectors 
disclosed as fig 6 above, one would then scan for example only the key areas, but one 
could scan them at a higher repetition rate or other ways effectively increasing the duty 
cycle for those regions and thus the perceived light in them. 

[00834] As noted in copending applications, the invention can be used as well for 
home control applications, for example of ones range, microwave, fridge, washer, dryer, 
stereo, TV etc. This desirably shares economies of scale with the automotive 
application, as well as potentially its control layout which allows ease of operation of 
both car and home, since the knowledge of its use is also shared with historic devices, 
particularly washers, dryers, and ranges almost all of which have knob based control 
panels. 

[00835] The screen of the invention too can show TV programs, especially nice for 
a control mounted in the Kitchen as it likely would be. With a lens /mirror change a 
projector version of the invention can even be reconfigured to project large screen TV 
images on a wall of an adjacent family room for example. . . c 

[00836] The video display can also serve to see live video of baby's room or front 
door (assuming cameras are placed there and properly interfaced by wireless or other 
means), access the internet (also using the computer in the invention, see general 
purpose discussion below) , wireless video feeds from cell phones, etc. 
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[00837] Where the computer used in the RTD invention for image processing and 
display is a general purpose processor (e.g. Intel Pentium 4 based) and software (such 
as a Microsoft Windows) , it is possible for the computer to be used for general use in 
the home, or in the car when stopped, and the system is easily integrated with other 
optional hardware or software from others. And because the machine vision processing 
is include to solve for knob or finger location, it is possible to use the image processing 
function to solve for other events as well. This is especially true, since the demands for 
control per se using turning of physical details or finger touch are relatively infrequent. 
[00838] For example, It can be used to monitor baby position, in the car or in the 
home. It can be used for a variety of vehicular tasks such as lane following , 
observation of cars on the right or left and their position, and so forth. All this is possible 
as the vision is essentially free, if one has it there anyway and provides it in general 
purpose form. 

[00839] In this situation, where processing capability is being shared, it is 
desirable to prioritize the tasks to be performed, giving priority to acting on the drivers 
commands, and safety issues such as imminent crash threats exterior to the vehicle, 
with relatively lesser priority to those not related to vehicle control or crash situations, 
such as monitoring persons or objects in the vehicle 

[00840] Interestingly, many useful functions which use processing power available 
"for free" can be accomplished in periods which control issues normally addressed by 
the RTD Touch or Physical details are not required. 

[00841] One such optional machine vision processing use mentioned in previous 
applications has been for backing up. A task which can have high priority as it is 
stressful, but only lasts a short while. It is unlikely any other control functions are 
required while doing this. The vision processor can find the edges of vehicles or other 
objects potentially in the way, and particularly where stereoscopic cameras are 
employed, determine in real time the distances to tjiem as an aid to the driver. This is 
much more data than possible with simple ultrasonic sensors used for backing warnings 
today, and in addition can show on the video display, where the problem is. Conversely, 
as also noted previously, the driver can touch the problem portion of the image he sees 
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on his backing up view displayed and the camera and vision processing portion of the 
invention can continually monitor that location. 

[00842] Another example, is at intersections, where cameras located near the 
headlights for example and pointed outwardly from the vehicle to the right and left can 
be used to look for likely signatures of vehicles in motion, and a signal, such as a sound 
or light given when such motion exists (the absence of motion meaning the cars 
entering the intersection from the right or left have come to a stop, or there are no cars 
on the intersecting road in the vicinity) 

[00843] As a lower priority item, but one to be undertaken constantly , the Machine 
vision processing of the invention can also monitor switch locations of controls in the 
front or back seat areas if cameras are placed there for observing same. 
[00844] It should be noted that the big horizontal display space in the region at the 
top portion of the instrument panel afforded invention , is optimal for providing two side 
by side images. In the case of the intersection , it can be from the two cameras one in 
each direction. Or it can be from one camera and some data concerning the image or 
some completely separate data, both displayed at once- a big advantage. 
[00845] Another point is that because the screen may be a touch screen, the user 
may touch an object image displayed (eg a car ahead) or with a finger gesture he can 
quickly draw a circle around the object, which data can be sensed and the appropriate 
action taken, such as locking the system on that image to track it (and the object it 
represents) relative to the users car. 

[00846] It is noted that a general purpose vision processor to share tasks can be 
provided independent of the Instrument panel tasks. In addition another display than 
that of the RTD can be used to display video data or information. However when all 
these functions are combined, maximum cost effectiveness results. 
[00847] On another note, it has been estimated that 60% of jthe vehicles on the 
highway have only the driver in them. It would thus be safe to say that 80% or more, 
have only the driver and the passenger in the right front seat. The typical provision of 
the vents high in the center stack, in the middle of the dash, is likely to provide the best 
air distribution to the 20% or less of the passengers in the rear seat, as well as fresh air 
to the face of the driver and passenger, thought this is also achievable with vents in 
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other places. And these vents for the purpose of rear seat passengers could be even 
less required, given the trend toward special rear compartment vents in cars having 
rear seats.(today nearly all higher class vehicles have this now) 
[00848] So then is it necessary to have vents high in the center stack? They are 
there to give best distribution at lowest cost, and to allow the driver and passenger to 
direct them on their face. But I feel they are also there because there is no really 
valuable thing to put in their place. In other words to make having an alternative 
arrangement valuable enough to compensate for the relatively small disadvantage 
entailed. 

[00849] I feel that the invention changes this equation. The valuable thing for both 
safety and convenience to provide an interactive display, such as that of the invention, 
in the drivers best line of sight high on the centerstack /dash . The invention is not 
limited to this arrangement, however it is thought that location of the vents to the sides, 
or to the bottom, or somewhere else entirely (e.g. a user controllable pop-up vent on the 
top of the dash) frees up this very valuable space. 

[00850] It also should be noted that for a larger instrument panel display (e.g. 1 2- 
15 inches diagonal, assuming a pure rectangular format, which it doesn't have to be), 
only two technologies seem today to cost effectively apply ( neither one of which is the 
conventional LCD Flat Panel Display used for navigational displays and the like and 
generally 5-7 inches diagonal today). The first is rear projection on to a simple passive 
screen/control surface, in which a small low cost display chip so to speak with a large 
number of pixels is magnified by projection optics to fill the screen, or alternatively one 
or more beams (of different colors) is swept raster fashion by a simple xy mirror 
scanning device which itself may be a semiconductor or other chip (e.g. a Microvision 
company MEMS type). Both types used for rear projection can use solid state light 
sources, such as lasers or LEDs, particularly if a wide range of color fidelity is not 
required, the case in many control system applications using the invention. 
[00851] The second method thought to be economic in the future, will be Organic 
light emitting diode displays (OLEDs) which may contain a large number of low cost 
addressable emitting elements. Here again, we can expect to employ this technology 
sooner if color fidelity is not needed. 
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[00852] It should be noted that for near term use in automobile instrument panels, 
full color presentation is desirable, but not required. The same holds true for high 
display resolution. The invention can use monochrome, two color , three color or any 
combination of colors including full renditions, as the financial justification of the 
application allows. This issue is particularly of interest as one considers ultra long 
lifetime solid state sources which today may not have high grade color rendition 
[00853] For most automotive control applications studies by transportation 
researchers suggest that lettering for controls in the center stack more or less on the 
drivers line of sight, should at least 5 mm x 5mm per letter, and larger perhaps for 
elderly drivers and others who benefit by it. For a resolution density of 5x 5 pixels (25 
total )per character for example, 5 mm high letters would thus mean only 250x250 
pixels need to be projected for a 250x250mm (10x10inch) display. This is easily 
achieved with even modest display technology today. Of course, higher resolution 
allows more dense displays, for various navigational data and other images that could 
be desirable. 

[00854] The invention allows the commonly used controls, like the climate control 
or radio to be utilized in a conventional manner, compliant with FMVSS 101 . But the 
space is shared with other control functions. In addition, a large display is provided in 
addition, which serves as well as an auxiliary control surface to have virtual controls 
[00855] The large display surface also allows one to have some windows type 
control screens side by side, rather than sequentially addressed by a joystick. And one 
can have them operated by touch as opposed to with a joystick selection as well, as 
used in the BMW l-Drive This is much more intuitive and safer, and quicker. 
[00856] BMW makes claim that a large number of things aren't done while driving, 
and thus you don't need to have a very intuitive means of easily performing them . But I 
take issue with this. Many people on road trips would like to make a small change to 
occupy their time on the road productively. 

Figure 26 

[00857] Figure 26 illustrates an advantageous arrangement of the RTD in a sloped 
instrument panel control and display surface 2610 of the invention including power 
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operated vents 261 1 and control knobs 2612 and 2613. In this illustration, the display 
/control surface is approximately 13 inches high and 10 inches wide . The 13 inch 
height extends up to the point int the instrument panel so as to use the space commonly 
occupied by vents in many vehicles. 

[00858] For those vehicles which have the dash sloping away as shown (to give a 
desirable feeling of extra space) it may be difficult for a belted driver to reach at the 
uppermost point. Thus the display in this region would be primarily for display rather 
than control purposes. 

[00859] A set of one or more vents 261 1 are located near the base of the 
windshield in the sloping away portion of the instrument panel. While mechanical 
cables can be used with driver controlled levers for example to actuate them, in the 
example shown the vents are activated via electric motors which may be controlled for 
example using control features on the RTD surface to save cost. Such vents can be 
provided in such a way as to sweep back and forth or up and down automatically to aid 
air distribution. Or they may also move to desired memorized preset situations for 
different drivers or passengers, also adding perceived value to the vehicle. 
[00860] By providing vents in such a way, the key area of the instrument panel is 
reserved for important display of data, and the ability to act on the data , for example 
when data is touched in a relevant location using the invention to confirm or 
acknowledge or otherwise act on the data. 

[00861] Also shown in figure 26 is a front view of an example of an instrument 
panel area (shown in dotted lines) which extends approximately 1 - 2 inches to the left 
of where the aforementioned RTD would be. This can be achieved for example, if the 
steering wheel 2640 is made smaller in diameter, allowing one to see and actuate 
controls such as virtual or physical buttons in the extended region 2670 shown in 
dotted lines. The steering wheel of smaller diameter may itself be enabled by providing 
the speedometer and other typical instrument cluster gages on the surface of the RTD 
itself, thus making it unnecessary to look thru the steering wheel at them ( a factor 
which requires larger diameter steering wheels). 

[00862] While it can be appreciated that the display and control surface 261 0 can 
extend further down, The display at the bottom is shown to approximately end at the 
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point at which vision is difficult without taking eyes off the road, and as well in some 
vehicles the point at which a gear lever can be in the way. 

[00863] Finally a note on force feedback. The Immersion Corp patent on which 
the l-drive is apparently based discloses a force feedback function to the person 
operating the device as, in effect, a glorified mouse. This motorized detent function 
feedback is the big feature distinguishing this control from those before it. But in 
reading over 20 different descriptions of l-drive motoring experiences in the automotive 
press, I have yet to hear one mention of this feature. In other words, it has apparently 
brought little or no utility to the new or casual user of the system. 
[00864] The instant invention also has the ability to provide force feedback to the 
user also disclosed in copending applications. The force function is most simply 
applied thru the screen to a finger touching the screen, fingers turning a knob, or sliding 
a slider or touching a rocker switch or what ever. We thus need to ask, will these 
features be also ignored in the press (and thus of little utility to the motoring public) and 
if so why? 

[00865] I don't think this will be the case for the following reason. The force 
feedback , vibratory or otherwise, is generally applied such that it can be felt at the 
point at which the person is physically touching something that displays directly what it 
is. Thus the force function in the instant invention serves primarily to reinforce a quick 
glance, though in some cases the control display could be used entirely by feel. This 
seems to be quite different than the purpose of l-Drive force feedback, which apparently 
is to reinforce something you already know, (e.g. that you are turning something, or 
selecting a menu category found by moving the stick to the right). In the primary 
application of the instant case, we are glancing briefly and then using the force 
feedback to assist us by taking over the function if needed. 

[00866] It should also be noted that the scanning unit of fig 20 has a large depth of 
field, and together with a convex screen shape, it is possible to project sufficiently 
focused information on this screen over a wide angle from a short distance behind the 
screen, from a position toward the center of the arc of the screen, often without the 
need for mirrors and other elements. This simplifies the system , reducing depth and 
lowers costs. Costs are also reduced with convex display surfaces as expensive fresnel 
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lenses are not required to collimate incident radiation in the situation where limited 
angles of acceptance of screen diffusing material (such as 3M Vikuti) are used The 
screen can even have as noted a variation in depth in the direction toward the driver, 
which might be a jutting out ridge, or an indented large oval screen or whatever. This 
allows a great deal of stylistic variation in the display and control features provided the 
automobile manufacturer or user. 

[00867] In some cases it is also possible to project data over a wide angle on a 
convex surface, not just using a flying spot projector scanner of fig 20, but from whole 
field light valve based projectors such as LCD, LCOS or DLP chips for example, 
[00868] I feel the RTD invention herein can save cost it is felt even over 
conventional instrument panels today, as well as the reconfigurable ones having 
displays with which it directly competes. This is because, for example, it allows one to 
replace some of the switch gear, wiring, and other components of the vehicle, reduce 
the complexity of the installation, and minimizes design and retooling costs. 
[00869] The elimination of components also can result in a weight saving of the 
vehicle, further reducing cost . 

[00870] In situations of high sun load for example, When the camera or other 
detector of the invention senses increased radiation on the screen from inside the 
passenger compartment of the vehicle, the display intensity can be turned up, or added 
illumination sources switched in to augment the normal one. Or the display can 
purposely illuminate the region of the screen with high sun (or other ambient light 
source) load more than others. This can be done for example with a flying spot scanner 
such as fig 20 by just scanning that region 3 times, for every one scan of another region 
say. Or by controlling intensity as one scans. Or by adding the light of other sources. 

! 

! 

Figure 2 7 ] 

[00871] Because the screen can be irregular jin shape, it is possible, as shown in ■ _ . . 
figure 27a to have air vents such as 2701 and 2702 at the sides of the screen/control 
surface 2710, which can fit around the vents, thus conserving space. Alternatively or in 
addition, the screen can wrap around vents located at the bottom or top or other 
locations. This arrangement also allows one to label the vent itself in a programmable 
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way. For example showing high - for high velocity air from the vent, or showing the 
vent direction. And the temperature from the vent (which might be sensed individually 
at the vent) could be shown as well. Such capability is not present in current vehicles, 
and could be a selling feature. 

[00872] In some cases it may be desirable to have air distribution vents located on 
the screen itself, which is made possible in the invention as shown in fig 27b. As shown 
light from the projection source 2740 passes thru the rear surface 2741 of clear plastic 
vent plenum 2750 and out through the screen 2755. Alternatively the plenum can be in 
front of the screen as shown in dotted lines 2770. Where vent louvers are desired, 
illustrated by a second such plenum 2770 having louvers 2771 (emitting air blast 2775) 
in the section of the drawing, the projector would generally be programmed to project 
no light there. Otherwise the projected image passes thru the plenum and is observed 
by the driver on the other side. 

[00873] Because of air condensation possibly on the plenum walls it may not in 
some cases be possible to project high resolution data in this region. Version 2760 can 
also work with LCD or other flat panel displays, with the plenum attached in front 
thereof. But these generally do not allow projection thru to a scattering surface which 
can be on the plenum front itself, such as surface 2761 . Where this is done, the 
scattering material of the screen in the region of the plenum 2745 is deleted such that 
light passes through to the front to be scattered by surface 2761 . 
[00874] Figure 27 also illustrates another point namely that one can have a slot or 
other shaped cutout 2720 in the screen, in to which is mounted a device such as CD 
Player 2725, into which a user can insert data storage media for example . The player 
can be mounted right to the screen as shown if a sufficiently sturdy screen is provided. 
Fastening means such as bolts 2726 and 2727 may for example be used. In this 
situation, it may be desirable to mount the device as shown in dotted lines 2726, at an 
angle alpha to the normal to the screen surface such that its side\|alls are moreprjess 
parallel to the diverging projection radiation 2728, in order that obscuration of the 
projection by the cross section of the player is minimized. Just as in the vent case, 
one can project data 2730 concerning the media or the player state right next to the 
player on the screen. 
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Figure 28 

[00875] Figure 28 illustrates further TV camera based applications also including 
near IR light sources, which may if desired, use the computer processor of the RTD. In 
many countries some critical markings are poorly illuminated or of minimal contrast and 
cause difficulty when driving. Or in other cases it is desired to rapidly see a sign, for 
example of a store while driving down an expressway at high speed. 
[00876] Consider for example the name of a street 2800, etched in stone on a 
house 2801on a street comer in Italy. Typically there is low contrast of the street 
marking and it is poorly lighted. When driving in traffic at night such a marking is often 
impossible to see while keeping ones eyes on the road. 

[00877] Using the invention, herein a camera 2820 connected for example to the 
vision processor 2821 of the RTD (or another suitable processor) is aimed outward from 
the side of the car 2825. And if desired, another camera (not shown) is aimed looking 
outward from the other side of the vehicle. A sufficiently powerful Near IR LED light 
source 2840 (which may be strobed and synchronized with the camera if desired) 
illuminate the house markings up to the second story, (they are usually on a wall just 
below the second story) . As the car travels down the street, the camera continues to 
read the images coming in (say every 0.05 second) and character recognition software 
such as that provided by Cognex company in Natick, Mass., determines the name of the 
street it finds and displays it on the screen of the rtd or vocally annunciates it) . If 
sufficient light power, and slow enough car speed, the data can be gathered on the fly 
without blurring beyond the ability of the software to determine the name. 
[00878] The picture of the street sign can also be displayed on the RTD screen 
2850 in the vehicle. This also provides a manual back up in case the system can 
recognize the existence of the sign (and cue the RTD to display it), but the software 
cannot discern the alphabetic characters. The sign image can be held in memory as 
well to give the driver a much longer time to view the image than he would other wise 
have had using eyesight alone. In the simplest case one can use only a visually 
displayed image, but if this is the case one would have to display a stream of images 
which could be distracting to the driver when in motion. 
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[00879] To aid in such activity it would be helpful to have retro-reflective street 
signs (or other markings as desired). This would vastly increase the ability to make 
such readings quickly and with certainty that it was a sign of interest being read. Color 
or other types of codes can be used as well to indicate this. Many signs on freeways 
are made in this way now and can also be read in this manner, through the cameras 
would need to be pointed ahead. 

[00880] Also shown in figure 28 is the addition of a one or more additional 
cameras such as camera 2860 looking forward into the path of the vehicle over a wide 
angle. Typically, but not necessarily , this camera might be reasonably close and 
parallel to the drivers line of sight when driving normally down the road, such that the 
display of the cameras image would be centered on the display in this manner as well. 
This camera may have IR bandpass filter 2861 in front of it (as could 2820) in order to 
preferentially accept IR Radiation. In this case however, it is the radiation from two 
sources external to the vehicle, namely the IR headlamps (or Tail lamps) such as 2870 
of oncoming cars or cars ahead and the IR radiation of stationary "light houses" such as 
2880 located along the road. 2880 for example could be near a railway crossing and 
either always on, or caused to be illuminated, perhaps with a 5 HZ flashing light, when 
a train is coming. Or it could be at an intersection when cars coming are sensed. There 
are a great many examples where this could be valuable, and it does not require radio 
communication to the vehicle (such as a wireless intelligent highway proposed warning 
device would). 

[00881] The RTD not only contains the ability to process the images so obtained to 
aid the driver, but to also display them in a size large enough to be useful. The display 
screen at night, might normally be programmed to be black in a certain region which the 
driver could easily see, in the absence of any IR of the set wavelength being acquired 
by the camera . Repetition rate or other criteria could also be screened by the RTDs 
processor if it was desired to discriminate the inputs further (eg a headlight on 
continuously, versus a lighthouse near a crossing blinking at 5HZ.): When an IR video 
image meeting the criteria set was obtained, this would be immediately displayed in the 
window of the screen. This image would stand out, especially at night, and be itself like 
a warning light. But if the image had meaning, for example flashing (at a repetition rate 
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lower than the camera frame rate typically) or moving up or down, or if it was of a 
certain shape, or if it was obviously coming closer, then the driver would also be more 
informed. And he also can immediately see the relative position of the image on the 
screen, relative to his vehicle. . And this image could be acquired from a long distance 
thanks to the camera sensitivity and the IR source power (which, like 2880, can be high 
without blinding anyone in the visible) . This gives an added margin of safety over what 
is possible today. 

[00882] From a military view point it is noted that an interesting feature of the 
invention is that the core elements - computer, projector and camera (or other sensing 
device) may all be in one small, armor plated box. The screen and physical controls if 
hit by shrapnel or the like can easily be replaced or fabricated locally. This is 
particularly interesting as the screen size increases in vehicles or other weapons 
systems. An alternative LCD or plasma screen of large size is difficult to protect . In 
addition the screen of the invention can be of bullet proof transparent material, and the 
knobs etc can be Kevlar if desired. 

[00883] It should also be noted that the display of the invention can be entirely in 
the near infrared which allows it to present no visible target to enemy fire, but may be 
operated by troops having night vision goggles or other equipment on. The printing size 
on t he screen can be increased in this situation if the goggle resolution is lacking 
(going back to normal size, when the display is switched back to the visible, easily 
accomplished with IR and Visible lasers such as in the display of figure 20). As noted 
the invention is excellent for allowing maximum operation efficiency in stressful 
situations needing quick reaction, possibly to large amounts of data. It is both intuitive 
and easy to see and work. 

[00884] The driver passenger in the front seats of a vehicle are in relatively fixed 
positions, it is possible to generate on the screen of the invention an autostereoscopic 
image using lenticular or other techniques known in the art which can be viewed by the 
driver in such a manner as to provide more information to the driver, and/or to allow 
controls of the invention which can be sensed for example in various z axis positions 
into the screen, to be used to control 3-D data images on the screen as well as 
controlled functions of the vehicle. 
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[00885] It is also noted that reflection of light from datums (also called targets or 
markers if put on specially for this purpose) on knobs and sliders and other physical 
details has largely been described herein as the means for determining their linear or 
rotational position with respect to a fixed point on the control surface, or with respect to 
the camera or other sensing device used to determine datum location. . However points 
on knobs for example detected by the camera are typically reflective, but can be self- 
luminous. This can be done either by providing electrical wiring or fiber optic wiring to 
the knob to do so, or by using light from illuminated nearby portions of the screen to flow 
into the knob, and back to the camera. This can be done by designing the knob out of 
plastic for example having suitable optics within it to cause this to occur. In other words 
the projector projects on the screen, (or through the screen directly, if no scattering 
material is in the path under the knob portion in question) and this light is effectively re 
routed in the knob such that the re-emitted portion can be viewed by the camera. 
[00886] This is an alternative to direct reflection of light which could also be from 
the projector, back to the camera and can be useful in some instances. 

[00887] FMVSS 101 is contained in USA CFR Title 49 part 571 

[00888] "Light" as used herein includes all electro-magnetic wavelengths from 

ultraviolet to near infrared. 

[00889] The foregoing discussion should be understood as illustrative and should 
not be considered to be limiting in any sense. While this invention has been particularly 
[00890] shown and described with references to preferred embodiments thereof, it 
will be understood by those skilled in the art that various changes in form and details 
maybe made therein without departing from the spirit and scope of the invention . 
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